DataJoint is a validated Built-on partner in the Databricks Brickbuilder Partner Network (Bronze tier)

Book a Discovery

  • Every result leaves a trail

    Every result leaves a trail


    Most labs rely on results that nobody can fully explain. The number is in a paper, a slide deck, or a decision that has already been made. Ask which inputs, which version of the code, and which run produced it, and often no one can say. This first episode is about that problem:

    a result that is still in use after the record of how it was made has been lost.

    A number is not a result

    A result is a number plus its provenance: the inputs it came from, the code that computed it, the parameters used, and the run that produced it. Without those, you have a value and no way to check it. A lot of lab data is in this state, in spreadsheet cells and in folders named final_final_v2.

    This works until someone asks how the number was produced. That might be an auditor, a regulator, a reviewer, or an AI system that reads your data and acts on it, which is more and more common.

    If the record was never kept, there is nothing to show them. Usually nobody made a mistake. The tools they used simply didn’t record how each result was made.

    The record has to come from the work itself

    The usual fix is discipline: label everything, keep the notebook current, name files consistently. That holds until a busy week, a quick re-run, or a handoff to someone new. Provenance that depends on people remembering is missing at the moment it’s needed.

    In a grounded workflow, each step records how its result was made at the moment it runs: what it used, what code ran, and what it produced. There is no separate documentation task. The design of the workflow is the operation, and every step writes its own record.

    Ask how a result was made, and look it up

    Once the record is automatic, the question that used to stall a meeting becomes a lookup. You can trace a result back through the analysis to the raw data, the instrument, and its origin. You can ask which results depend on an input that just changed.

    A new scientist, an auditor, or an AI agent can follow any number back to the moment it was made, without needing the one person who happened to remember.

    That is the difference between a result you report and one you can defend. The first asks for trust. The second shows the record.

    Every result leaves a trail. The work is to build the workflow that keeps it.

    Justin Tripodi

    , , ,

  • Where does your provenance stop?

    Where does your provenance stop?

    THE SHORT ANSWER

    Automatic provenance covers what a system performs itself. At the boundary with instruments, vendor files and manual steps, the system cannot describe what happened because it did not do it. From that point on, provenance depends on a person keeping a commitment, and the strongest organizations know exactly where that line is.

    The more useful question is where it stops

    Provenance conversations are usually about coverage. How much do you have, how far back does it go. A more revealing question is where it ends.

    A system can record every transformation it performed, because it performed them. No one has to remember, and nothing depends on discipline. That part is strong.

    What a system can describe, and where that stops.

    Then the work reaches a boundary. An instrument writes a vendor file. Someone runs a step by hand. A value passes through a spreadsheet on its way somewhere else. At that boundary the system stops being able to describe what happened, because it did not do it. A system cannot infer what it did not compute.

    What sits on the other side

    When I ask teams what happens between the instrument and the point where the platform picks the data up, the answer tends to be a version of the same thing: that part is a convention. Everyone knows to name the files correctly.

    ‘Everyone knows’ is not a record. It is a practice.

    A practice works for exactly as long as the people who hold it are the people doing the work. That is a shorter horizon than most coverage claims imply, and it is rarely written down anywhere that would survive a reorganization.

    Boundaries are not the problem

    Every real research environment has them, and any vendor claiming otherwise is describing a diagram rather than a laboratory. The argument is not for eliminating boundaries. It is for knowing precisely where yours fall.

    The organizations in the strongest position are not the ones claiming end-to-end coverage. They are the ones who can point at the boundary and name who is accountable on the other side of it.

    The strongest version goes one step further

    Knowing where the boundary falls is the minimum. The better arrangement is a system that refuses to accept data across that boundary without a source record attached, so a write that arrives with nothing is not a tolerated gap but a failed load.

    That turns the boundary from a place where discipline is required into a place where discipline is enforced, a materially different guarantee when the person who held the discipline leaves.

    The readiness question

    Where does your provenance stop being automatic and start being a convention?

    If nobody can point to the line, the line is still there. It is just being crossed without anyone accounting for it.

    Justin Tripodi

    , , ,