DataJoint is a validated Built-on partner in the Databricks Brickbuilder Partner Network (Bronze tier)

Book a Discovery

  • Which result do you trust?

    Which result do you trust?

    Biomedical researchers struggle to reproduce their own work. In a 2024 survey of 1,630 of them, published in PLOS Biology, 72% said their field faces a reproducibility crisis. Of those who had tried to repeat one of their own studies, 43% could not.

    That last number is the stranger one. Reproducing someone else’s work means a different lab, different hands and different reagents. Reproducing your own work removes most of those differences, and it still fails. You run the same analysis a second time, get a different number, and no one can say why. The first run is already in the paper. Now you have two results and no way to tell which one is right.

    This kind of failure is quieter than fraud and does more damage, because nothing flags it. A result you cannot account for looks the same as one you can, until someone asks you to account for it.

    Why results diverge

    A result can change between runs for reasons that never show up in the analysis: a parameter set by hand and never written down, a library that updated, a random seed, a different machine. A reviewer would not catch any of these, because the analysis looks the same both times.

    So running it again does not settle anything. A third run may give a third number. Without a record of what each run used, you are comparing results you cannot explain.

    The fix is a record of every run

    It would be easy to claim that the same inputs always give the same answer. That would not be true. Much of what changes a result sits outside the code: software versions, seeds, hardware. No workflow can keep all of that fixed.

    What a workflow can do is keep the record. In a grounded workflow, every run stores exactly what produced it: the inputs, the code version and the parameters. When two results differ, you put the two runs side by side and see what changed, such as a dependency or a parameter. You can also see which run matches the paper.

    This is the lineage claim from the first episode, applied to a second run, and it holds without conditions. The number may still move. The record tells you why it moved and which version to trust.

    Which one do you report?

    The runs do not all have to agree, but when they disagree you need to be able to explain why. Then when a reviewer, an auditor or your future self asks which result is correct, the answer is in the record and not in someone’s memory.

    A result you can account for is a result you can defend. A result you cannot account for is a liability wearing a lab coat.

    With the record, you do not have to guess which result to trust.

    Sources

    Cobey, K. D., et al. (2024). Biomedical researchers’ perspectives on the reproducibility of research. PLOS Biology, 22(11), e3002870.

    Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022). A large-scale study on research code quality and execution. Scientific Data, 9, 60.

    Justin Tripodi

    , , ,

  • Every result leaves a trail

    Every result leaves a trail


    Most labs rely on results that nobody can fully explain. The number is in a paper, a slide deck, or a decision that has already been made. Ask which inputs, which version of the code, and which run produced it, and often no one can say. This first episode is about that problem:

    a result that is still in use after the record of how it was made has been lost.

    A number is not a result

    A result is a number plus its provenance: the inputs it came from, the code that computed it, the parameters used, and the run that produced it. Without those, you have a value and no way to check it. A lot of lab data is in this state, in spreadsheet cells and in folders named final_final_v2.

    This works until someone asks how the number was produced. That might be an auditor, a regulator, a reviewer, or an AI system that reads your data and acts on it, which is more and more common.

    If the record was never kept, there is nothing to show them. Usually nobody made a mistake. The tools they used simply didn’t record how each result was made.

    The record has to come from the work itself

    The usual fix is discipline: label everything, keep the notebook current, name files consistently. That holds until a busy week, a quick re-run, or a handoff to someone new. Provenance that depends on people remembering is missing at the moment it’s needed.

    In a grounded workflow, each step records how its result was made at the moment it runs: what it used, what code ran, and what it produced. There is no separate documentation task. The design of the workflow is the operation, and every step writes its own record.

    Ask how a result was made, and look it up

    Once the record is automatic, the question that used to stall a meeting becomes a lookup. You can trace a result back through the analysis to the raw data, the instrument, and its origin. You can ask which results depend on an input that just changed.

    A new scientist, an auditor, or an AI agent can follow any number back to the moment it was made, without needing the one person who happened to remember.

    That is the difference between a result you report and one you can defend. The first asks for trust. The second shows the record.

    Every result leaves a trail. The work is to build the workflow that keeps it.

    Justin Tripodi

    , , ,