DataJoint is a validated Built-on partner in the Databricks Brickbuilder Partner Network (Bronze tier)

Where does your provenance stop?

Automatic provenance covers what a system performs itself. At the boundary with instruments, vendor files and manual steps, the system cannot describe what happened because it did not do it. From that point on, provenance depends on a person keeping a commitment, and the strongest organizations know exactly where that line is.

The more useful question is where it stops

Provenance conversations are usually about coverage. How much do you have, how far back does it go. A more revealing question is where it ends.

A system can record every transformation it performed, because it performed them. No one has to remember, and nothing depends on discipline. That part is strong.

What a system can describe, and where that stops.

Then the work reaches a boundary. An instrument writes a vendor file. Someone runs a step by hand. A value passes through a spreadsheet on its way somewhere else. At that boundary the system stops being able to describe what happened, because it did not do it. A system cannot infer what it did not compute.

What sits on the other side

When I ask teams what happens between the instrument and the point where the platform picks the data up, the answer tends to be a version of the same thing: that part is a convention. Everyone knows to name the files correctly.

‘Everyone knows’ is not a record. It is a practice.

A practice works for exactly as long as the people who hold it are the people doing the work. That is a shorter horizon than most coverage claims imply, and it is rarely written down anywhere that would survive a reorganization.

Boundaries are not the problem

Every real research environment has them, and any vendor claiming otherwise is describing a diagram rather than a laboratory. The argument is not for eliminating boundaries. It is for knowing precisely where yours fall.

The organizations in the strongest position are not the ones claiming end-to-end coverage. They are the ones who can point at the boundary and name who is accountable on the other side of it.

The strongest version goes one step further

Knowing where the boundary falls is the minimum. The better arrangement is a system that refuses to accept data across that boundary without a source record attached, so a write that arrives with nothing is not a tolerated gap but a failed load.

That turns the boundary from a place where discipline is required into a place where discipline is enforced, a materially different guarantee when the person who held the discipline leaves.

Where does your provenance stop being automatic and start being a convention?

If nobody can point to the line, the line is still there. It is just being crossed without anyone accounting for it.