DataJoint is a validated Built-on partner in the Databricks Brickbuilder Partner Network (Bronze tier)

Book a Discovery

  • Can you trust it?

    Can you trust it?

    THE SHORT ANSWER

    Reproducibility asks whether you can rebuild a result. Trust asks something you face far more often: can you rely on a result you did not produce yourself? A teammate’s, an agent’s, last quarter’s from someone who has since moved on. The weak answer is that you trust whoever made it. The strong answer is that you do not have to, because the record lets you check it.

    The result you didn’t make

    Reproducibility is about your own work. Trust is about everyone else’s. In production, most of the results that cross your desk are not yours. You did not run them. So the question that governs your day is no longer whether you can rebuild your own work. It is whether you can rely on work you did not do.

    Trusting the person, or trusting the record

    There are two ways to trust a result. You trust whoever produced it, or you can check it yourself. The first feels efficient, and it is a liability: it cannot be checked, and it walks out the door when that person does. The second is the only kind of trust that survives turnover and scale.

    Trust that depends on who made the result does not scale. Trust that lives in the record does.

    Figure · Two ways to trust a result

    Agents make this urgent

    A proof of concept has a human in the loop who vouches for the output. You trust the result because you trust the person standing next to it. Production does not work that way. Agents produce results faster than anyone can personally verify, and the person accountable for a result did not watch it being made. You cannot trust that volume by trusting an author. You can only trust what each result carries with it.

    A result that carries its own evidence

    You can lean on review and reputation: they are good, so I trust them. It helps, and it decays, because it rests on the person you are trying not to depend on. The stronger arrangement is a result that arrives with its inputs, its code, and the workflow that produced it, so anyone, or any agent, can check it without asking whoever made it.

    That works only when the record is not something the author kept by hand. The result cannot exist without its inputs and the code that computed it, so the record cannot drift from the result. Trust stops being a relationship. It becomes a property of the result.

    The readiness question

    The last result you acted on that you did not produce yourself: did you trust it because you could check it, or because of who handed it to you?

    Justin Tripodi

    , ,

  • Can you rerun it?

    Can you rerun it?

    THE SHORT ANSWER

     Reproducibility is the real test of whether scientific data is ready. Not whether a result was written down, but whether it can be rebuilt on demand; by someone else, after the people and the systems have moved on. Most teams can show a result. Fewer can run it again. That gap is where AI validation and regulatory review tend to break.

    The request that separates companies

    In an evaluation I eventually ask for the same thing. Show me last year’s best result. Then rerun it, start to finish, while I watch.

    Most teams can produce the result. The figure, the readout, the number that went into a decision. Fewer can rebuild it. Not because anyone was careless, but because the result was never designed to be produced a second time.

    A reproducible result is a different object

    A result you cannot reproduce is not a weaker version of one you can. It is a different kind of object. It records that something happened once. It cannot promise it would happen again

    The question is not whether it happened. It is whether it can happen again.

    That is not a quality gap you close with a better report. The report describes the result. Reproducibility is a property of how the result was made.

    Not a niche worry

    In 2026 the Brazilian Reproducibility Initiative, more than 50 labs working together, took a set of published biomedical experiments and ran them again. Depending on the criteria, 20% to 44% reproduced. Fewer than half. 

    This was not a survey or an estimate. It was the experiments, run a second time, by independent labs with nothing to sell. And a 2024 survey in PLOS Biology found that 72% of biomedical researchers already agree there is a reproducibility crisis. The people doing the work are not surprised.

    The part I sit with is not the number. It is why so much of it did not rebuild. The initiative traced much of the gap to methods that were written down but never written to be run again: steps left ambiguous, details that lived in one person’s hands. 

    Records were kept, protocols were written, reports were filed. The work still did not rebuild. Documentation was never the missing piece.

    Why ready and rerunnable are the same question

    A result reruns only when four things are recoverable together: the inputs, the code, the environment they ran in, and the order they ran in. Lose any one and you can still describe what happened. You cannot reproduce it.

    Most systems capture the first and narrate the rest. That is enough to pass an audit of the record. It is not enough to rebuild the result, which is a different and harder guarantee.

    What a vendor will not tell you

    Here is the concession. Most existing results will never be made rerunnable, and chasing them usually is not worth it. You do not go back and rebuild five years of work. You draw a line. You make everything after it reproducible by default, and you are clear about which side of the line a given result sits on

    The stronger version does not rely on discipline

    Knowing the line is the minimum. The better arrangement is a system that will not record a result it cannot rebuild, so reproducibility stops being a practice a person maintains and becomes a property of the record itself. That difference holds up when the person who maintained the practice leaves.

    The readiness question

    The last time you were asked to reproduce a result, could you, or did you have to piece it back together?

    Surfacing that in a room costs an afternoon. Discovering it inside an AI validation costs the result, and every decision that leaned on it.

    Sources

    The Brazilian Reproducibility Initiative, “Estimating the replicability of Brazilian biomedical science,” eLife, reviewed preprint (2026)

    Cobey et al., “Biomedical researchers’ perspectives on the reproducibility of research,” PLOS Biology 22(11):e3002870 (2024)

    Justin Tripodi

    , , ,

  • Where does your provenance stop?

    Where does your provenance stop?

    THE SHORT ANSWER

    Automatic provenance covers what a system performs itself. At the boundary with instruments, vendor files and manual steps, the system cannot describe what happened because it did not do it. From that point on, provenance depends on a person keeping a commitment, and the strongest organizations know exactly where that line is.

    The more useful question is where it stops

    Provenance conversations are usually about coverage. How much do you have, how far back does it go. A more revealing question is where it ends.

    A system can record every transformation it performed, because it performed them. No one has to remember, and nothing depends on discipline. That part is strong.

    What a system can describe, and where that stops.

    Then the work reaches a boundary. An instrument writes a vendor file. Someone runs a step by hand. A value passes through a spreadsheet on its way somewhere else. At that boundary the system stops being able to describe what happened, because it did not do it. A system cannot infer what it did not compute.

    What sits on the other side

    When I ask teams what happens between the instrument and the point where the platform picks the data up, the answer tends to be a version of the same thing: that part is a convention. Everyone knows to name the files correctly.

    ‘Everyone knows’ is not a record. It is a practice.

    A practice works for exactly as long as the people who hold it are the people doing the work. That is a shorter horizon than most coverage claims imply, and it is rarely written down anywhere that would survive a reorganization.

    Boundaries are not the problem

    Every real research environment has them, and any vendor claiming otherwise is describing a diagram rather than a laboratory. The argument is not for eliminating boundaries. It is for knowing precisely where yours fall.

    The organizations in the strongest position are not the ones claiming end-to-end coverage. They are the ones who can point at the boundary and name who is accountable on the other side of it.

    The strongest version goes one step further

    Knowing where the boundary falls is the minimum. The better arrangement is a system that refuses to accept data across that boundary without a source record attached, so a write that arrives with nothing is not a tolerated gap but a failed load.

    That turns the boundary from a place where discipline is required into a place where discipline is enforced, a materially different guarantee when the person who held the discipline leaves.

    The readiness question

    Where does your provenance stop being automatic and start being a convention?

    If nobody can point to the line, the line is still there. It is just being crossed without anyone accounting for it.

    Justin Tripodi

    , , ,