A catalog describes data and is maintained separately from it, so it can drift out of agreement with what it describes and nothing breaks. A structure constrains how the work runs, so it cannot disagree with the data it holds. Documentation is forensic; structure is preventive. Most organizations have invested heavily in the first.
A catalog describes. A structure constrains.
This distinction decides more evaluations than it gets credit for, and it is usually discussed as a tooling preference rather than the difference in guarantee that it is.
A catalog is written alongside the data and maintained by a separate process. It can drift, and when it drifts nothing breaks. The entry still reads correctly. It is simply no longer true.
A structure cannot drift, because work cannot run in a shape the structure does not permit. It does not record what somebody believed. It reflects what the system did.
Two ways of knowing what your data is, and what each one guarantees.
How you find out a description has gone wrong
Teams show me catalogs covering thousands of datasets, carefully maintained, impressive to walk through. When I ask how they would know if an entry had become wrong, the answer is a version of the same thing: someone would notice, eventually, while using the data.
That is not a criticism of those teams. It is the honest answer, and it is the answer almost everywhere, because a description has no mechanism for reporting its own accuracy. Drift is silent by construction.
Forensic and preventive are different jobs
Documentation is consulted after something has gone wrong, and its value is that it helps you piece together what happened. Structure’s value is that a class of problem cannot occur.
Documentation is forensic. Structure is preventive.
Most organizations have invested heavily in the forensic half and call the result governance. It is worth having. It is not the same as knowing, and the distinction tends to surface at the least convenient moment.
Why AI changes the stakes
A drifted catalog used to mislead one person at a time, slowly, with a human in the loop who might notice something odd. A system reading that catalog at scale will repeat the error consistently and present it with the same confidence as everything else it says.
The readiness question
If a description of your data became wrong tomorrow, what would tell you?
If the answer is a person eventually noticing, that is worth knowing before an AI system starts reading it.
AI adoption across biotech and pharma drops sharply in the areas closest to discovery decisions. The reason is not resistance to change. A scientist who acts on an output they cannot trace or rerun is absorbing risk the organization has not priced; so refusing is the correct professional judgment.
A line worth pushing on
The line I keep hearing this year, in some version, from every corner of the industry: we no longer have a data problem. We have a confidence problem.
Half the industry now says a version of that, which usually means the phrase has stopped doing work. So the better question is where the confidence problem lives, because it is not evenly distributed.
It lives at the bench
Across biotech and biopharma organizations, reported AI adoption reaches 42% for generative molecular design, 40% for biomarker analysis and 29% for ADME prediction, as reported by Drug Discovery News in February 2026.
Benchling 2026 Biotech AI Report, a November 2025 survey of ~104 biotech and biopharma organizations. Reported by Drug Discovery News.
Those are the places where discovery decisions get made. Adoption falls as the stakes of the decision rise, which is the opposite of what a technology adoption curve normally looks like.
Refusing is the correct professional judgment
The standard reading of that data is scientist resistance, and it leads to more training and more change management. I think the standard reading is wrong.
Declining to act on an output you cannot reproduce is not caution. It is the job.
A scientist who acts on a confident answer they cannot reproduce is not being efficient. They are absorbing risk the organization has not priced and will not acknowledge until something goes wrong. Declining is not conservatism. It is the job.
Confidence is a property of the system, not a state of mind
Which means it cannot be trained into people. It comes from the same inputs and the same code returning the same result, every time, with a record of how it got there. When that is true, adoption follows without a campaign. When it is not, no amount of enthusiasm from leadership will move the number.
This is also why the confidence problem and the data problem are not separate. Confidence is what a well-built foundation feels like from the outside.
The question to take into your next AI review
Would your most careful scientist stake a program decision on this output? If not, what would have to be true?
The second sentence is the useful one. Whatever she names is your roadmap, and it will be shorter and more specific than any maturity assessment you could buy.
Scientific data becomes AI-ready in one of two ways: as a consequence of how the work runs, or by being described after the work is finished. On paper the two look identical. The guarantees are not, because a retrofit depends on someone correctly describing afterward what happened, and it is never finished while new work keeps arriving.
Two routes to the same document
Scientific data becomes AI-ready in one of two ways, and the resulting documents can look nearly identical. What they guarantee could not be more different.
The first route makes readiness a consequence of how the work runs. Structure is enforced as the work happens, and provenance falls out of execution instead of being a separate task somebody owns.
The second attaches readiness to work that has already finished. Someone goes back, describes what happened, and records it.
The same four stages. Two different points at which readiness is established.
Why a retrofit does not stay finished
When I ask teams deep into a retrofit what happens to work that ran before the project started, the answer is prioritization. When I ask what happens to the work running now, the pause is longer.
That is the whole problem in one exchange. A race against production, and production does not pause.
It is not an argument that retrofit fails. Retrofit works. What it cannot do is stay finished, because the process that created the backlog is still running.
Retrofit is often the only honest option
Nobody rebuilds twenty years of research because a strategy deck asked them to. For historical work, describing what happened is frequently the only route available, and doing it carefully is real work worth funding.
The distinction that matters is between choosing retrofit for the past and choosing it for the future. The second choice usually gets made without anyone noticing it was a choice.
Retrofit works. What it cannot do is stay finished.
What the two approaches guarantee
Work that produces itself in a ready state carries a record of what the system did. Work made ready afterward carries a record of what somebody believed the system did. Both are useful. Only one of them cannot disagree with the work it describes.
Data that was made ready afterward carries one more assumption: that whoever did the describing understood what happened. Sometimes they did. The work that produces itself in a ready state does not need the assumption.
The readiness question
For the work your team ran last quarter, did readiness happen as part of the work or after it?
If the answer is different for last quarter than it will be for next quarter, that is progress. If it is the same, the retrofit is not a transition. It is the operating model.
Most enterprise AI programs cannot say what they will change or by when. The metrics that prove one in R&D are not AI metrics at all. They are R&D metrics: cycle time from data to decision, the share of results that can be rerun without their author, and the time it takes to answer an audit question.
The old question is retired
Two years ago, nearly every leadership conversation I was in started the same way: do you have an AI strategy? That question has been answered everywhere, usually more than once. The one that replaced it is harder. What does AI change, and by when?
Does it shorten the path to a candidate? Does it improve the quality of a go or no-go decision? Does it reduce the rework nobody budgets for? Or is it a feature?
The number behind the shift
MIT NANDA found that 95% of organizations saw no measurable P&L impact from an estimated $30 to $40 billion in enterprise GenAI spend.
MIT NANDA, “The GenAI Divide: State of AI in Business 2025,” July 2025.
Pharma is not exempt, and the R&D leaders I talk with know it. The internal conversation has moved from ambition to accounting. I read that as healthy, not pessimistic.
The budget asymmetry nobody names
Here is the uncomfortable part. Most AI budgets fund what the model gives you. Almost nothing funds what makes the answer worth trusting.
Most AI budgets fund what the model gives you. Almost nothing funds what makes the answer worth trusting.
That asymmetry is why a well-funded program can produce a technically correct output nobody will act on. The money went to the half of the problem that demos well.
The metrics that prove an AI program are not AI metrics
They are R&D metrics. Cycle time from data to decision. The share of results that can be rerun without the original author. Time to answer an audit question. AI either moves those numbers or it does not, and all three can be measured this quarter without a new dashboard.
The question I now hear from finance, in one form or another: if this were a drug program, what would the readout be? Most AI proposals do not have an answer. The ones that do tend to be the proposals worth funding.
The question to take into your next AI review
If your AI program were a clinical program, what is its primary endpoint?
If naming it is difficult, the program may still be worth running. It is not yet worth defending, and at some point this year someone will ask you to defend it.
Sources
MIT NANDA, “The GenAI Divide: State of AI in Business 2025,” July 2025. Coverage: Forbes, 26 Aug 2025.
AI-ready is not a single property. It usually means at least four different things depending on who is asked: stored and documented, accessible without friction, reproducible by a scientist, or defensible to an auditor. Organizations get into trouble when they buy against one definition and are later judged against another.
A question with four correct answers
Every organization I talk to wants AI-ready data. I hear it in first meetings, in RFPs, in board decks. What I almost never hear is what ready means. And when I do receive a definition, it rarely survives the next person in the room.
Ask whether the data is AI-ready and the yes comes fast. Ask what ready means and you get the pause. I have sat through that pause enough times to know what follows: the same four answers, from four different chairs in the same meeting.
The same question, asked of four roles in one organization.
The head of data says it is in the warehouse and the schema is documented. R&D IT says accessible through an API without raising a ticket. A scientist says they can reproduce a result and see how it was produced. The compliance lead says they can show an auditor where a number came from and who changed it.
All four are correct. None of them is the same test.
Why the ambiguity is expensive
This is where it gets expensive, and I have watched it happen more than once. A company buys against one definition and gets judged eighteen months later against another. The tool did what it promised. The promise answered a question nobody had confirmed was the right one.
Gartner predicted that through 2026, 60% of AI projects unsupported by AI-ready data would be abandoned — a figure surveyed across industries, not pharma alone. My guess, and it is only a guess: a meaningful share of those projects involved data somebody inside the organization had already certified as ready. Certified against one of the four definitions. Abandoned against another.
The four definitions are not bought together
This is the part that catches even careful buyers. Findable does not mean reproducible. Accessible does not mean defensible. Each definition implies different work, different systems, different owners. A purchase that satisfies one can leave the other three exactly where they were.
None of the four is wrong. The mistake is treating them as one word.
What to do about it before the next evaluation
My advice is not glamorous. Ask the four people separately, in writing, before anyone drafts a requirements document. The disagreement is the requirements document. Finding it in a room costs an afternoon. Discovering it in year two costs the program.
The readiness question
If you asked four people on your team what AI-ready means, how many answers would you get?
If the answer is one, that is worth verifying rather than celebrating.
Sources
Gartner, “Lack of AI-Ready Data Puts AI Projects at Risk,” 26 February 2025.gartner.com Primary disclosure naming the analyst, survey period and sample size.
This is the written version of the talk I gave at Lab of the Future Europe in Amsterdam on 23 September 2026, in the AI to Innovate session. Sources, figure credits, and further reading are at the end.
Agents now do real scientific work, and the bottleneck has moved from what a model can do to what a team can check. Workflow grounding is DataJoint’s answer: agents read and extend the same declarations the work runs on, so every result traces to its inputs, its code, and the decisions behind it. A harness makes a model useful for one campaign. A formally defined workflow is what lets the work compound after the agents are gone.
The Lab of the Future Europe congress spent two days in Amsterdam on one question: what scientific work will look like when we make full use of automation and artificial intelligence to solve the hardest problems in medicine and life sciences. Automation, connected instruments, digital CMC, and AI running through all of it.
Instruments stream into shared stores instead of onto a workstation under the bench. Notebooks and inventory systems speak to each other. AI agents read a protocol, write the analysis for it, and call the tools to run it. In synthetic chemistry, propose–make–measure cycles close with nobody in the middle.
The possibilities feel intoxicating. Hand an agent a question and it proposes the work, runs what can be run against the data already in hand, and comes back with candidates ranked and its reasoning attached — for many questions at once.
At DataJoint we implement the scientific data foundation for research teams in pharmaceutical, clinical, and academic research. So the question we keep asking is whether the work compounds: does the group know more than it did, in a form the next question can use? That is a question about the workflow, not about the agents, and it gets harder to answer the longer the agents run before anybody asks it.
The bottleneck moved
Today, agents analyze the literature covering an entire domain. They propose and execute statistical analyses. They propose hypotheses, and they design experiments, protocols, and methods. It feels as though we should be able to solve problems that were out of reach, not only to work faster.
The progress has been rapid, and it has created new bottlenecks.
Ask a room of scientists what goes wrong when they work with capable agents, and the answer is almost never that the model was not clever enough. The most common complaint I hear is this: all I do is review what the AI produced, and I am behind.
Agents generate copious output that looks good. It reads as plausible, often as insightful and promising. And every so often there is an embarrassing, inexplicable mistake — the wrong data used, or two datasets silently mismatched.
That is the hard part. Errors you miss, inside work that looks right. How do we know which conclusions are trustworthy? And will we be able to defend them six months or a year from now?
Agents are wonderful brainstorming partners. Longer stretches of unattended work that lead to decisions need a much tighter level of rigor, and rigor is a property of the workflow rather than of the model.
How AI gets grounded
AI agents have improved in two ways: model capabilities and their grounding in the subject matter. Over three years the models gained larger context windows, better tuning, tool use, and reasoning they can explain.
The grounding is our ability to adapt a model to your subject matter, your questions, and your data. We have been through several rounds of this, each one tightening the grounding of an answer in the relevant data and making the answer easier to check against it.
Era
How knowledge is reached and checked
2021–23 · fine-tuning
absorbed into the weights — not checkable
2022–24 · prompting
pasted into the window — by reading it
2023–24 · retrieval
ranked passages — by following the citation
2025 · context engineering
assembled before the answer — by the context engineer
2025 · MCP
live query into your systems — by asking the system again
2026 · science harnesses
assembled per campaign — by the code and reasoning it records
now · workflow grounding
curated as the work runs — traced to inputs, code, decisions
The second column describes the checkability of the data used. A model weight cannot be asked where it came from. A citation points at a document but does not reproduce its derivation. A live query tells you what a system holds today, but not how it got there.
The newest of these is the science harness, a term now doing a lot of work in the field. A science harness is the environment built around a model that gives an agent tools, memory, compute, and a record of what it did.Kosmos, from Edison Scientific, and Claude Science, from Anthropic, are two of the best-known. They are serious engineering, and the record they keep is real evidence: it names what ran, on what, and when.
If we think of models as wild horses — strong, intelligent animals with minds of their own, entirely capable of acting destructively — then a harness is what lets them do useful work on the things you care about. But a harness is fitted to one horse for one race. What you also need is a track.
A track sets the lanes so that many horses can run at once, at full speed, with checkpoints along the way. And when you lay out a track, you make the straights long: long enough for an agent to do real work, to branch, to iterate on a design, and to come back with findings for a decision instead of questions along the way.
The earlier grounding methods wrapped around the model. They are model-centered approaches, and their aim is to maximize and showcase what a given model can do. A harness reaches your experimental data from the outside, and the one thing it cannot check is the thing you are judged on.
Workflow grounding
Workflow grounding starts from the other end: from the scientific workflow the lab actually performs. An agent is grounded in that workflow when it reads and extends the same declarations the work runs on — the entities, the dependencies between them, the code that produces each result, and the state of every derivation. The context is not assembled around the model; it is the workflow itself, and it outlasts the agent that used it. This is where much of the next few years of effort will go.
Other speakers at this congress described something close to it in their own words — multiagentic workspaces, and multi-agent orchestration as the layer that connects capabilities into coherent scientific workflows: the shared, persistent environment any number of agents plug into, defining the tools, the interfaces, the guardrails, and the rules for operation and record keeping. Different words, converging on the same conclusion: the durable object is the workflow, not the agent.
The self-driving-laboratory work arrives at it from synthetic chemistry rather than from data infrastructure: La Agente Óptima, from the Acceleration Consortium, separates the agent’s reasoning from the executed campaign and keeps a persistent optimization state between them, so that auditability follows from the architecture instead of from logging what the model said. (A survey of data agents published five days after this talk extends the term to a workflow harness, and lists among its open problems that what one run learns does not reach the next — which a harness cannot fix, because it dissolves with the run that used it.)
So how should we know whether any of this is working? Usually we measure how much faster we produce results with AI assistance. But the more important question is what is left at the end. After a year of such work, will you have a solid body of work that the next phase can build on? Or a year of copious documents that nobody — including you — can explain, defend, or reproduce?
It will not matter nearly as much how much work was produced, or how fast.
Precise notation lets knowledge compound
I have heard many people say that in 2026 the hottest programming language is plain English: whatever you can articulate clearly and precisely, your agent can deliver. There is something real in that, and there is a trap in it.
We may be living through a period much like how mathematics was done a thousand years ago. Arabic mathematicians in the Middle Ages could solve quadratic equations. But they did it, and taught it, in copious prose with diagrams. Here is a ninth-century solution to a quadratic equation, translated into English — about ninety words of correct reasoning that you can follow, and cannot transform.
Precise notation lets knowledge transform, compound, and be checked.
The notation that evolved over the following eight hundred years is far more than shorthand. It expanded our cognitive capacity to transform expressions and derive new results. The same equation now takes two lines, and schoolchildren learn it in every culture on earth.
Notation also makes work independent of the machinery that carries it. Music could not be expressed in prose. Before musical notation, music passed by ear from master to pupil — which is how a great many research teams still operate. Beethoven scored the Ninth in 1824 for valveless horns and trumpets, gut strings, and keyed wooden flutes. Within a generation of his death, valved brass was taking over from the horns and trumpets he wrote for, and tonight the same score is played on instruments he never heard. It never needed rewriting, because it specifies the music and not the machinery.
Research workflows need that same independence. The instrument is replaced, the cluster is decommissioned, the file format is superseded, the vendor is acquired — every one of them on a shorter cycle than the science. A workflow that exists only as the scripts that happen to run it expires with them. A workflow written in a formal notation moves, because what it specifies is the work rather than the machine that ran it.
Precise notation lets knowledge transform and compound.
Software has run this experiment too. In the 1960s and 70s it was thought that programming would become more accessible if programs read like English sentences, and that push produced COBOL and the fourth-generation languages. Computer scientists have since found algebraic notation the more potent form, and the English-like programming languages have faded. The exception is SQL, a relic of that era that survived because the vendors standardized it (ANSI in 1986, ISO in 1987), so a query written for one database runs on another with minor changes. People outside programming still use it every day: a formal language, scoped to one domain, that a person can read.
Notice where agentic AI has made its most impressive progress: exactly the domains that already had precise formal notation and verifiable results. Mathematics and programming. That is where a model can be trained against a verifiable answer under clear rules.
The same holds for research workflows. To invent a workflow, to transform it, and to judge whether it is any good, we need a way to express it precisely and formally — so that humans and machines both understand it unequivocally.
A formal language for research workflows
So what language do we use to express a research workflow and reason about it?
We already have several domain-specific languages, for data, for computation, for infrastructure, and for governance. A real workflow needs several of them at once, and each is correct inside its own scope.
Language
What it specifies, and what it leaves out
CWL, Nextflow, Snakemake
A directed acyclic graph of tasks with typed inputs and outputs — plainly, the steps and the order they run in. It leaves out the entities those steps produce: results are files the engine does not model, so nothing carries a result’s identity or its relation to the results on either side of it.
Terraform, Helm
A declarative target state for infrastructure, reconciled by a control loop — plainly, the machines and services, and what correct looks like. It leaves out the science: it can tell you a cluster is running, and nothing in it records what ran there.
Cedar, OPA
Authorization policy — principals, actions, resources, and conditions, evaluated per request; plainly, who may do what. It leaves out what the data is: a policy permits a read without saying what was read, or how that row came to exist.
dbt
SQL transformations with declared dependencies, materialized and tested inside a warehouse — plainly, how one table is built from other tables. It leaves out acquisition, instruments, computation that is not SQL, and work that is queued rather than done.
SQL DDL
Relations, domains, keys, and integrity constraints — plainly, the structure the database will hold. It leaves out how any of it came to be: derivation lives in code outside the schema, so a row cannot name what produced it.
Relational Workflow Model
Entities, their dependencies, the code on each, the constraints, and the state of every derivation, in one declaration carrying structure, dependency, and transformation together — plainly, the workflow itself, as a database that computes it. It leaves deployment and access control to the rows above, by design.
The lacks differ in detail and agree in kind: none of these languages carries the work itself. They describe a static structure, and scientific work is not static. It branches. It is rerun with a changed parameter. It is partly computed and partly waiting. In science, how a result was derived is the contract, and a specification that leaves derivation to an engine outside itself has left out the part that matters.
A traditional database records what happened. A computational database defines how it happens.
For this purpose we developed the Relational Workflow Model: a semantic refinement of the classical relational database that makes computational transformations a native part of the schema. It is the language for constructing and evolving a scientific data foundation, and we develop it as an open standard, so that projects built on it stay freely interchangeable.
The DataJoint library is the Apache-2.0 reference implementation. The argument for it as a formal language, worked through against a relational ERD of the same pipeline, is in Schema as Workflow Grammar.
The scientific data foundation
A formal way to express and transform workflows is what lets us define a data foundation in a real lab.
DataJoint provides the scientific data foundation where scientific work is codified as formally defined, self-executing workflows — free to evolve while their integrity, provenance, and reproducibility hold up by design — so that people and their AI agents can build on them with full trust.
It holds the typed entities and the dependencies between them, the code that reproduces each result, the integrity constraints, and the state of every derivation. It does that as a relational schema, and the schema is the specification of the workflow.
And it fits the infrastructure you already have: the instruments, the systems of record, the lakehouse, the compute. Instruments keep acquiring and systems of record keep their records; what arrives lands against a declared entity. The bytes stay in the object store you already run, because what the foundation holds is the entity, its lineage, and the code that produced it, not the payload.
The orange path on the right of that diagram is the only arrow a machine does not cross by itself. Agents read the schema and the job state, and their changes come back as declarations a person merges.
Using this model, people and agents alike can understand, express, extend, and transform the workflow as the project evolves.
The operational cycle
For the sake of this discussion, model scientific operations as a loop: you design the work, you operate it visibly, and you explain what you learned and how you know it.
Every group draws its own version of this loop, and nothing that follows depends on there being three acts.
Regulation has focused on the Operate side of this loop — that inputs and transformations are traced and the data well governed. That is why Operate is the one act carrying an audit mark. The loop works. It produced the medicines in current use.
What is not typically modeled or formalized is the context in the middle: the motivations and aims, the decisions, the observations. That happens in meetings, in emails, in conversations — and it is where a team’s knowledge accumulates. Just as with music before notation, it passes from master to pupil, between generations of researchers. It aggregates, but not in a form an agent can use to build the next question on.
Implementing the scientific data foundation lets us formalize the Design and Explain parts of the loop as well, so that every act becomes inspectable, auditable, and analyzable by humans and agents alike — not Operate alone.
But it also requires a process for a shared, managed, curated context. That context becomes the grounding for the research team and for the agents, and it is aggregated while the work happens rather than collated afterwards. Learning is branched, reviewed, and merged together with the workflow it belongs to — the git-based process, applied to what a project knows. Which means bringing that context out of the emails and the Zoom calls.
Observe and Decide were always happening anyway. Somebody always noticed what a run revealed; somebody always decided to stop or to go on. What is new is that both are written down, so they tighten the cycle instead of dispersing from it.
Each turn of the loop leaves the project knowing more than it did before.
This is what we are building at DataJoint: agents that contribute at every part of the loop, helping the team operate the cycle and aggregate the context while the work runs, rather than reanalyzing after the fact.
An example workflow
Here is a workflow expressed in the Relational Workflow Model. This is the schema diagram, where each node is a table representing a step.
The connections between nodes are foreign keys, and they establish both data integrity and execution order. Rows in those tables are the workflow artifacts produced at each step. Whereas a traditional database captures entities and the relationships between them, a workflow database defines and executes how the data is collected and transformed.
This one is liquid chromatography and mass spectrometry — twelve entities, public, and the same workhorse in discovery, in bioanalysis, and in QC, so the shape of it travels. It is at once a relational database and an executable workflow: data entered by hand or from the electronic laboratory notebook, parameters in lookup tables, ingestion recorded, computations logged and versioned. Each node is a snippet of Python that declares the data structure, the dependencies, the order of operations, and the transformation the step performs — one construct doing three jobs, which is why they cannot fall out of agreement. Computation is part of the schema definition, not something bolted on beside it. And the figure was not drawn: it is generated from the pipeline it describes, so it cannot drift from it.
One story about this pipeline. It is the exemplary design in our own preprint — and when we asked an agent to review it against our own design checklist, it found defects. A table linked to its upstream by a comment saying the keys matched, rather than by a foreign key that would enforce it. This made us both proud and embarrassed — an agent we coached outperformed our own reasoning. It is fixed now, in the public repository. And we expect this to become a virtuous cycle: improvements inform new improvements, applied automatically.
With a formal representation of the workflow, agents become remarkably effective at designing, evaluating, and operating research workflows — and, as on this pipeline, they begin to catch what the original authors missed, on exactly the dimensions we care most about: quality and rigor.
A production pipeline at Johns Hopkins
DataJoint was first battle-tested in neuroscience mega-projects. The MICrONS project characterized the structure and function of a cubic millimeter of mouse visual cortex — some 75,000 neurons imaged in the awake animal, and an electron-microscopy reconstruction of more than 200,000 cells and half a billion synapses — and DataJoint was the backbone of the shared neurophysiology component from 2016 to publication. The results filled a Nature special issue in April 2025; our own account of the pipeline underneath them is A Better Data Engine for Brain Science.
The pipeline below is from the Shuler Lab at Johns Hopkins — subject and colony management, photometry, electrophysiology with GPU computation, running in production for years.
The first thing the picture buys is that the whole operation is one legible object. Manual entry, acquisition, heavy computation, and reporting sit in one graph, so a question about any result is answered from the structure rather than from whoever happened to run it.
The second thing is that the schema follows the lab’s own code. When the code in their repository changes, the structure extends with it, so code and schema move together instead of drifting apart. Marshall Hussain Shuler, quoted in our April 2025 Nature Research Custom feature: “We have already saved months and ran experiments that we never could before.” Two different units in one sentence — months saved is throughput, and experiments that were not previously possible is feasibility, which no hourly rate prices.
Zoom into the electrophysiology chain and you can see the grain of it: the recording as it came off the probe, a clustering task carrying its parameter set, the clustering itself, and curation as its own declared step. Methods and their parameters are entities in their own right, not arguments buried in a script. So the pipeline keeps evolving as the lab takes on variations of the experiment: change a setting, run it again, and you get a second result beside the first rather than one that quietly replaced it.
Later in the pipeline the work becomes analysis, reports, and figures — and figures are computed through the same dependencies as everything else. A figure is normally the least governed thing a laboratory produces: made in a notebook, exported, pasted into a deck, and from that moment disconnected from the analysis it summarized. Here, recomputing the analysis recomputes the figure with it. A figure in a deck is an orphan, and in a regulated laboratory an orphan is an audit finding.
Trustworthy AI is achieved by distrusting AI
So how do we defend what we learn, when much of the work is proposed and performed automatically? Workflow grounding lets us turn agents loose on many things at once, which makes that question sharper.
Trustworthy has a technical definition now: the ability to meet stakeholders’ expectations in a verifiable way. That is ISO’s wording, in ISO/IEC TS 5723:2022, and the EU AI Act uses the same term. Verifiable — not persuasive, not well-documented, not confident.
Regulators on both sides of the Atlantic have written the same requirement into law and guidance. In the EU, the AI Act — Regulation (EU) 2024/1689 — names trustworthy AI as its purpose in Article 1, and for high-risk systems it requires automatic logging across the system’s lifetime (Article 12), information sufficient for a deployer to interpret the output (Article 13), and human oversight by people who can understand what the system did (Article 14).
Those high-risk obligations were then deferred by the Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since 27 July 2026 — to December 2027 for stand-alone Annex III systems, and to August 2028 for AI embedded in regulated products — while the Article 50 transparency duties took effect on schedule in August 2026. The date moved; the requirement did not.
Read them together and the same requirement appears in each: a record that lets someone else check the work. Not one of them asks whether the output was impressive.
And so trustworthy AI is achieved by distrusting AI — by making it show its work, and by making that work inspectable and verifiable. Verifiability, curation, and accountability follow.
Verifiability. A claim can be checked instead of believed, because the derivation behind it is recorded as structure and not as narrative.
Curation. The context is explicit, versioned, and owned by the project, rather than accumulated inside an agent’s memory. Agents come and go; the workflow and the curated context are what endure. If a new agent — or a human researcher — cannot reproduce the work from that context, the work is not finished. We do not forbid agents that remember. We refuse to depend on implicit memories.
Accountability. Agents can run long stretches autonomously, and people decide at checkpoints. Workflow grounding is what defines those stretches: an agent loops on a branch of the workflow, and at the end of the stretch its work is verified before it is merged. A checkpoint is not a rubber stamp — the accountable person has to understand what actually happened. And in each of these frameworks, it is the human scientists who are held accountable, because they have skin in the game. The AI co-scientist, well, has no skin.
None of that caps how much runs unattended. The gates sit on the consequential calls, and everything between them belongs to the agent. Explicit, inspectable aggregation is what makes agents faster, not what slows them down.
Ground for the next agents
Agents are fast and powerful, and we should assume they are ephemeral. Ground them in a workflow that is formally defined and written in one schema, and rigor holds while the agents run at speed.
This approach takes work. What it buys is work that aggregates and compounds — which is also what your regulators and your own quality people will ask you for.
AI agents will keep becoming more capable, and they cannot be the entire difference. The difference is what the operational cycle accumulates. The work in front of us is to ground the workflow, so that knowledge accumulates while the work runs.
The ideal agent is the one you never see. The ideal process lets the scientist see the science itself most clearly: what a decision implies, and what an experiment actually revealed.
What are your agents standing on, and where does their work go?
References and figure credits
Lab of the Future Europe 2026, Beurs van Berlage, Amsterdam, 22–23 September 2026 — lab-of-the-future.com/europe. The talk this piece is adapted from was given in the AI to Innovate session on 23 September.
How to Accelerate Research with AI Scientists and Multi-Agent Orchestration — Rob Brown, Sapio Sciences, Lab of the Future Europe, 22 September 2026. The published agenda describes multi-agent orchestration as the layer that connects these capabilities into coherent scientific workflows — a formulation close to the shared environment this essay calls workflow grounding.
Yatsenko, D., et al. DataJoint 2.0: The Relational Workflow Model. Preprint — arXiv:2602.16585. The formal definition of the model, its query algebra, and the substrate elements built on it.
Schema as Workflow Grammar, DataJoint, 12 August 2026 — datajoint.com/news/schema-as-workflow-grammar. The formal-language argument, worked against a relational ERD of the same pipeline.
ISO/IEC TS 5723:2022, Trustworthiness — Vocabulary, July 2022 — iso.org/standard/81608.html. The source of the definition of trustworthy used above: the ability to meet stakeholders’ expectations in a verifiable way.
Regulation (EU) 2024/1689, the AI Act, in force 1 August 2024 — eur-lex.europa.eu. Art. 1 (human-centric and trustworthy AI as the stated purpose), Art. 12 (record-keeping), Art. 13 (transparency and provision of information to deployers), Art. 14 (human oversight) for high-risk systems.
Regulation (EU) 2026/1744, the Digital Omnibus on AI, published 24 July 2026 and in force 27 July 2026 — eur-lex.europa.eu. Defers the high-risk obligations to 2 December 2027 for stand-alone Annex III systems and to 2 August 2028 for AI embedded in regulated products; the Art. 50 transparency duties applied from 2 August 2026, with a four-month grace period on the Art. 50(2) marking obligation for systems already on the market, ending 2 December 2026.
NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), 26 January 2023 — nist.gov/itl/ai-risk-management-framework, doi:10.6028/NIST.AI.100-1. The seven characteristics of trustworthy AI, including accountable and transparent and explainable and interpretable.
FDA, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products, draft guidance, January 2025 — fda.gov. A risk-based credibility assessment framework for an AI model in a stated context of use.
MICrONS Consortium, Functional connectomics spanning multiple areas of mouse visual cortex, Nature640, 435–447, 9 April 2025 — doi:10.1038/s41586-025-08790-w. The flagship paper of the April 2025 Nature special issue, The MICrONS Project. DataJoint was the data backbone of the neurophysiology component, selected in 2016.
A Better Data Engine for Brain Science, DataJoint, 22 April 2025 — datajoint.com/news/better-data-engine-for-brain-science. Our own account of the MICrONS pipeline and the labs that followed it — republished, slightly expanded, from the Nature Research Custom two-pager of April 2025, which is the source of the Marshall Hussain Shuler quotation in A production pipeline at Johns Hopkins (the two-pager as printed).
Müller, M., Bai, J., Gottstein, W., et al. La Agente Óptima: Towards Agentic Self-Driving Laboratories, 3 September 2026 — arXiv:2609.04564. LLM reasoning separated from the executed campaign, with a persistent optimization state between them, so that control returns to the agent only when progress needs interpretation. Evaluated on two physical platforms, including a five-day flow-chemistry campaign that raised yield from 30% to 59% over 23 experiments. Presented at this congress by Willi Gottstein of the Acceleration Consortium, in the Orchestrated Lab track on 22 September.
Kosmos: An AI Scientist for Autonomous Discovery, Edison Scientific, November 2025 — announcement, arXiv:2511.02824. A structured world model as the agents’ shared memory, assembled per run and dissolved with it: up to twelve hours, some 200 rollouts and 1,500 papers, every conclusion cited to code or to literature.
Zhou, H., Zhang, Y., Du, J., et al. Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses, 28 September 2026 — arXiv:2609.35255. Published five days after the talk, and cited here in the note above rather than in its argument. A survey organizing LLM data agents around five stages — perception, planning, execution, verification, repair — under what it calls the workflow harness, “the control structure that governs how this trajectory is constructed, monitored, and revised.” Its diagnosis matches the one above: “silent failures can persist even when individual components function correctly.” Two of its four open reliability problems — missing experience transfer, and the missing verification-repair repository — are the case for a durable object outside the run.
Claude Science, an AI workbench for scientists, Anthropic, 30 June 2026 — anthropic.com. Curated connectors to public reference databases, jobs submitted to the lab’s own HPC cluster over SSH, and an auditable history — exact code, environment, and message history — on every output.
Figures: the operational-cycle pair, the foundation in context, and the notation composite are adapted from the Lab of the Future deck; the LC-MS schema and the Shuler Lab pipeline are dj.Diagram output generated from the schema definitions in each pipeline’s public code repository; neither diagram draws on any data. The al-Khwārizmī plate is from Rosen’s 1831 translation of the Algebra and the Beethoven plate from the autograph of the Ninth; both are public domain.
THE SHORT ANSWER: The gap is whether the work can travel
Pharma R&D organizations share nearly identical AI ambitions, and the spending now matches the ambition. The gap is not strategy or talent. It is whether the work underneath can travel: whether a result can be rerun by someone who did not produce it, and whether a pilot that succeeded in one lab can move to the next.
Identical ambitions, identical budgets
I spend a lot of my week talking with pharma executives, and the ambitions have converged. Everyone wants the same four things.
Accelerate discovery.
Shorten development timelines.
Put AI in the hands of scientists.
Do it without adding risk.
The spending now matches the ambition. In Benchling’s 2026 Biotech AI Report, a survey of biopharma organizations reported by Drug Discovery News in February 2026, 80% of organizations plan to increase their AI budgets in the next twelve months.
Read that number the way I did. If nearly everyone has the same goals, and nearly everyone is raising spend, then neither goals nor spend will decide who wins. Something else will, and it is the one thing that never appears in a strategy deck: where each company is starting from.
Two companies, same strategy
Here is the contrast I keep running into. One company spent six years making every result rerunnable before it bought a single AI license. Another is modernizing in flight, with thirty active programs depending on the systems being replaced. Both call themselves AI-forward. Both are telling the truth.
But ask each one what a new hire would need to reproduce last year’s most important result. Only one has an answer.
Neither the goals nor the budgets will decide who wins. The starting line will.
The gap shows up eighteen months later
The gap never shows up in the pilot. Pilots are run by motivated people with the builder one desk away. It shows up eighteen months later, when the work has to travel: to another team, another site, or simply to next year.
The same report puts a number on why: 55% of organizations name data quality and availability as the number one reason AI pilots fail. Not model choice. Not talent. The condition of the scientific work underneath, and that condition was set years before anyone approved an AI budget.
Benchling 2026 Biotech AI Report, reported by Drug Discovery News, February 2026.
Preparation is quiet, and it decides the outcome
I have never seen any of it in a press release. Consistent data models. Provenance captured as the work happens, not rebuilt afterward. Workflows that produce the same answer twice. Quiet work. And it decides whether an AI investment compounds or evaporates.
The question to take into your next AI review
If this program had to move to another team or another site tomorrow, what would break first?
Whatever you name is the starting line. Not the strategy, not the budget. The thing you just named.
THE SHORT ANSWER: Most R&D organizations preserve their data and lose their reasoning. The measurements survive; why a target was dropped or a protocol changed usually does not. AI exposes the gap immediately, because a model can summarize what was kept and cannot recover what was never captured.
Data survives. Reasoning does not.
Most R&D organizations I speak with do not have a data shortage. Storage is cheap, instruments are prolific, and the measurements are almost always there somewhere.
What is missing is why. Why a target was dropped in 2023. Why an assay protocol changed between two campaigns. Which result convinced a team to move forward, and which one told them to stop.
What a typical R&D record retains, and what it does not.
The pattern repeats across organizations of every size: the compound history is in the system, and the decision history is in a few people’s heads. By the time anyone goes looking, one of them has usually left.
AI makes the gap visible immediately
This was survivable when the person who ran the work was the person who explained it. AI removes that arrangement. Ask a model why a program was killed and you will usually get an answer, fluently, assembled from whatever documents happen to have survived.
A fluent answer is not the same as a true answer.
The output is not obviously wrong, which is the problem. A plausible account built from a partial record arrives with the same confidence as a retrieved fact, and nothing in the answer tells you which one you got.
The negative result is the costliest thing you are not keeping
In discovery, the study that did not work is the least likely to be written up with its full context, the least likely to be reviewed, and the first to be forgotten when the person who ran it moves on.
It is also the result that narrows the search space for everyone who comes after. The principle I keep coming back to:
No result should be wasted, including the ones that failed.
Every negative result that is not preserved is a study somebody else will run again, at full cost, to reach the same conclusion.
The economics of this are documented. In PLOS Biology, Freedman, Cockburn and Simcoe estimated that more than half of US preclinical research cannot be reproduced, at a cost of roughly $28 billion a year in the United States alone.
Their breakdown of causes:
biological reagents and reference materials at 36%
study design at 28%
data analysis and reporting at 25%
laboratory protocols at 11%.
More than a third of the problem traces to how work is analyzed, reported and documented, not to the materials or the design.
The same paper notes that the underlying biases can result in entire studies never being published or reported at all. The negative result does not just lose its context. It disappears.
A recollection is not a record
Some organizations try to close the gap afterward: exit interviews before a scientist leaves, retrospective write-ups, a wiki that nobody maintains past the second quarter. Reasoning captured afterward is a different artifact. It is a recollection, which is useful and is not the same as a record.
A record is made at the moment of the decision, by the people inside it, with the alternatives still on the table. A recollection is what someone believes happened, filtered through everything that came after, including how it turned out.
That is why this does not fix itself later. The reasons get captured as the work happens, or they get approximated afterward. And an approximation is exactly what a model will treat as ground truth.
The question to take into your next AI review
Where does your organization keep its reasons, and who could find them without asking a person?
The second half of that question is the one that matters. If the answer is a person, the reasoning is not stored. It is borrowed, on terms that end when they do.
Sources
Freedman LP, Cockburn IM, Simcoe TS, “The Economics of Reproducibility in Preclinical Research,” PLOS Biology 13(6), 9 June 2015. journals.plos.org
Science, “Study claims $28 billion a year spent on irreproducible biomedical research,” 9 June 2015.science.org
THE SHORT ANSWER: A year ago, pharma R&D leaders evaluated AI by asking about the model. Now they ask whether a recommendation can still be supported six months after it is made. That shift explains why so many AI programs stall. The model was rarely the constraint. The systems underneath were never built to answer the second question.
The question used to be about the model
A year ago, most AI conversations in pharma R&D opened the same way. Which model? How large? Trained on what? The technology was new enough to be the subject.
That has changed faster than almost anything I have seen in this industry. In the portfolio reviews and steering committees I hear about now, the model barely comes up. What comes up is what happens afterward.
The top-20 pharma leaders I interact with have converged on what I believe is the more important question:
If we pursue a particular AI strategy, can we support our reasoning six months from now?
Not at the demo. Six months out, when a program decision is examined by a partner, a regulator, or whoever inherits the work. Not because the models failed. Because nobody could fully stand behind the output.
Why the shift is happening now
The timing is not a coincidence. Enough programs have reached the point where early AI decisions carry consequences, and the people who made them have moved on.
Gartner put numbers on the gap in February 2025. A July 2024 survey of 1,203 data management leaders found that 63% of organizations either lack the right data management practices for AI or are unsure whether they have them. On that basis, Gartner predicted that through 2026, organizations would abandon 60% of AI projects unsupported by AI-ready data.
Read the prediction carefully, because the wording carries the argument. It does not say the models underperform. It says projects get abandoned, which is a different kind of failure. Somebody could not stand behind the output when it mattered.
And abandonment is no longer hypothetical.S&P Global’s Voice of the Enterprise: AI & Machine Learning, Use Cases 2025 survey of 1,006 IT and line-of-business professionals across North America and Europe found the share of companies abandoning most of their AI initiatives before they reach production rose from 17% to 42% year over year, with an average of 46% of projects scrapped between proof of concept and broad adoption.
What walking it back involves
The question sounds simple until a team tries it. Take one AI-informed recommendation from last quarter and follow it backwards.
There are five steps.
Name the exact version of the code that produced the figure.
Retrieve the inputs as they existed then, not as they exist now.
Rerun it. Same number?
Show what changed in between, and who changed it.
Find the context that turned that number into a decision.
Most organizations manage the first two steps in an afternoon and stall on the third. Almost none reach the last one, because the context was never written down anywhere that outlives the person who did the thinking.
These are risk questions, not model questions
Can we explain how the recommendation was reached? Can we rerun it and get the same answer? Can we scale it without scaling our exposure? None of those are answered by choosing a better model, and all of them are being asked of leaders who inherited an AI budget without inheriting the systems underneath it.
That is an uncomfortable position, and it is where a lot of R&D leadership currently sits. The mandate arrived first. The foundation it depends on was built for an earlier era, when the person who ran the analysis was also the person who explained it; and was still down the hall when the questions came.
What the organizations making progress have in common
It is not budget. The teams I see making steady progress are rarely the ones spending the most. What they share is a decision made early, before anyone was watching:
That trust, governance, scientific rigor and full context are foundational rather than paperwork arranged around the real work.
That decision costs real money in year one and compounds in value every year after. It rarely appears in a strategy deck. Leaders make it quietly, and the results speak later.
The question to take into your next AI review
Which AI-informed decision from last quarter could you walk back to the raw data today, step by step?
If the answer takes more than an afternoon to establish, that is the finding. Not a failure of the model, and not a reason to slow down. It is a measure of the distance between where your AI program is and where it will need to be six months from now.
Sources
Gartner, “Lack of AI-Ready Data Puts AI Projects at Risk,” 26 February 2025.gartner.com Primary disclosure naming the analyst, survey period and sample size. Both figures appear verbatim.
S&P Global Market Intelligence, Voice of the Enterprise 2025. spgglobal.com The primary source: a survey of 1,006 midlevel and senior IT and line-of-business professionals across North America and Europe. Both figures appear in S&P’s own wording.
Run “data integrity,” “data consistency,” “data governance,” and “data provenance” through Google’s Ngram corpus, and ask when each of them entered use.
The four terms arrive in order. Integrity comes first, rising from the 1970s — the decade the relational model arrived — and dominating the vocabulary for the next four decades. Consistency follows in the 1980s and settles early at a modest level, where it stays. Governance and provenance don’t launch until the mid-2000s. Then governance climbs steeply enough to very nearly catch integrity by 2022, while provenance lifts only faintly: in the most recent data it still trails integrity by a factor of eight.
The relational model accounts for that gap. Codd and fifty years of database research talk about integrity and almost never about provenance, because the model solved the problem at a deeper level. Provenance became a named discipline in the 2000s in workflow and e-science computing (Buneman 2001; the W3C PROV working group from 2011), where the substrate could not enforce relationships between artifacts. If your results are files produced by scripts, nothing stops you from holding a result whose inputs are gone. So you keep a record beside the data describing what happened, and you work hard to keep that record honest.
Provenance is the complete record of a data entity’s origin: what produced it, from what inputs, under what conditions, and through what chain of transformations.
Customers routinely expect that definition to cover two things it leaves out: the scientist’s reasoning, and governance.
Provenance records the how, and stops short of the why. The standards ask who, when, and by what means, and so does 21 CFR Part 11. Someone who asks for provenance and means “capture why the scientist chose this analysis” is asking for a different system. Say so early.
Governance is the wider regime — access, cataloging, audit, retention — and provenance is one component of it. DataJoint produces the authoritative provenance that a governance system consumes.
The literature also splits provenance into prospective, the declared plan, and retrospective, the observed run (Freire 2008; Lim 2010). Integrity is preventive. Provenance tooling is forensic. Prevention and forensics are different guarantees, and much of the confusion in this field comes from buying one and assuming the other.sic. Prevention and forensics are different guarantees, and much of the confusion in this field comes from buying one and assuming the other.
The Standards Landscape
Three families of systems account for most of what the field means by provenance tooling. They are also how the rest of the world will phrase its questions.
W3C PROV is the vocabulary: Entities (a thing), Activities (something acting on entities over time), Agents (who bears responsibility), joined by relations like used, wasGeneratedBy, wasDerivedFrom, and wasAttributedTo.
Figure: the PROV core types and relations, from the W3C PROV Primer (W3C Working GroupNote, 30 April 2013), reproduced under the W3C Document License.
OpenLineage is the same shape recast for data engineering — runs of jobs that read and write datasets — and it is the lingua franca of Airflow, Spark, and Marquez. Governance catalogs — Unity Catalog, Collibra, Alation, Purview, DataHub — record table and column lineage across the enterprise.
Then there is compliance, which is where pharma actually lives. ALCOA+ — attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, available — and 21 CFR Part 11, with audit trails that no one, administrators included, is permitted to delete. ALCOA+ is defined in PIC/S PI 041-1 and in the UK MHRA’s GXP data integrity guidance; Part 11 is 21 CFR Part 11, Electronic Records; Electronic Signatures. These are obligations of a deployment and a process, never features of a library.
In genomics the same shape appears again in the workflow standards of the Global Alliance for Genomics and Health (GA4GH), and in data science in the run-metadata stores that record inputs, parameters, and outputs for each run.
What every one of these families has in common is that the record is kept beside the data. Keeping it there is the only option available when the substrate cannot do better. The consequence is that there are now two things to keep in agreement, and nothing but diligence keeps them there.
One Slice Each
Three approaches cover most of what pharma runs today, and each covers its slice well. Document versioning tracks records rather than computation: version every controlled document, with no computational lineage underneath it. That is the quality and regulatory world, where Veeva Vault is the de facto standard. Metadata cataloging renders enterprise lineage as a graph, as Collibra and Informatica do; it traces, it does not enforce. Domain audit logging captures every edit with user, timestamp, and reason inside one boundary, which is what Medidata Rave does for electronic data capture in clinical trials. Large internal integration platforms move research data at enormous scale, with provenance supplied by whatever tooling sits on top of them.
Each covers one slice, well. None makes provenance a property of the data itself. The provenance system and the data system are separate systems, and separate systems can diverge.
How the ELN and the Workflow Fit Together
The system a bench scientist actually touches is the electronic lab notebook (ELN): Signals Notebook across most of large pharma, Benchling across much of the adjacent market. It is where the provenance chain starts, and the two systems fit together at that seam. The notebook attests to what was done at the bench. The workflow carries what was computed from it.
Every generation of the notebook, from paper through the AI-assisted tools now arriving, rests on the same mechanism: an experiment is recorded, and a second person reviews it and signs.
That signature is provenance of a specific kind: attestation. A named person asserts that the work was done as described, which is what ALCOA+ codifies and why contemporaneous is on that list. Recorded as the work happens, not reconstructed afterward. It is the right instrument for the bench, where what has to be vouched for is human judgment and physical work. It says someone vouched. It does not say the result could not have been otherwise.
Vince Paton, who spent years selling these systems, draws the line: “An ELN captures the experiment, DataJoint captures the science.” The notebook is authoritative for what a person did at the bench. The workflow is authoritative for what was computed from it. Neither is trying to be the other, and the two kinds of assurance compose: attestation at the boundary, structure downstream of it.
So the integration is a join, not a migration. The notebook’s entry identifiers and sign-off enter the workflow as the extrinsic record on the rows they describe — source, agent, time — and every result computed from those rows carries that identity in its lineage. Ask where a figure came from and the chain runs from the result, through the computation that produced it, back to the signed notebook entry it began in.
The join has to work by reference rather than by copy, and that is the practical requirement customers state themselves. Pointing an ELN’s search at a lakehouse or an object store would mean replicating that data into its backend, which they decline to do. Nothing needs replicating. The ELN keeps its record, the workflow keeps its structure, and the provenance chain crosses the boundary intact.
Two Pictures: Assembled vs Declared
Compare what each approach actually produces.
First, from the W3C PROV Primer, the complete provenance graph for one newspaper article: every entity, activity, agent, and relation needed to describe how a single artifact came to be.
Figure: the complete provenance graph for the Primer’s worked example, from the W3C PROV Primer §5, reproduced under the W3C Document License.
It is assembled by observation, it is stored next to the thing it describes, and keeping it accurate is somebody’s job.
Now the schema of a liquid chromatography–mass spectrometry (LC-MS) workflow: twelve entities and their parts, dependencies running left to right, origins carried by shape and color.
That one diagram is the lineage of every row in every table. Nobody assembled it. It was declared before any data existed, and the database refuses to hold a computed result whose upstream dependencies are not present and correct.
A catalog can disagree with the data it describes; a schema can’t.
Why can a catalog drift when a schema cannot? Because the two artifacts in the second picture — the data and its lineage — are not two artifacts. They are the same one.
Provenance: Extrinsic vs Intrinsic
This is the distinction the rest of the vocabulary hangs from.
Extrinsic
Intrinsic
Source of guarantee
Observational: recorded as metadata
Structural: entailed by the data model
Posture
Forensic, after the fact
Preventive, before the fact
Failure mode
Can be incomplete, or drift
Cannot be created broken
Answers
“This datum was reported to come from there”
“This result must derive from these inputs”
As used here the terms are specific to the Relational Workflow Model, the model DataJoint introduces, formally defined in Yatsenko & Nguyen 2026. That is the definition this essay argues from. Intrinsic means entailed by the schema and enforced by referential integrity, not merely inherent in some general sense.
Extrinsic provenance is a record captured about data whose origin lies outside the model’s structural guarantees: read from an instrument, transcribed from a notebook, pulled from a laboratory information management system or a sample registry. Every system in the preceding sections produces this kind, and produces it well. Intrinsic provenance is entailed by the structure of the Relational Workflow Model and maintained by the database itself. It is not a record kept about the data. It is the data model.
Mapped onto the literature: extrinsic corresponds to retrospective provenance, intrinsic to prospective, with one sharpening. In DataJoint the prospective structure is enforced by referential integrity, so it is a binding guarantee rather than a declared intention.
The pair of terms came to DataJoint from Doug Welsh, in February 2026, from the commercial side rather than the engineering one and new enough to still hear the vocabulary the way a customer hears it. Two claims were sharing one word: customers who said they already had provenance had a careful record of what had happened, and DataJoint was offering something that is not a record at all.
The available reading was that DataJoint keeps a better record than the alternatives. He took the harder one: that it keeps no record, and the guarantee comes from somewhere else.
That reading was available to him because he had spent a career selling the alternatives: data virtualization into pharmaceutical accounts, the data lake and its catalog, unified storage, array stores for scientific data. Every one of those systems keeps a record about data it does not control, which is the extrinsic model sold for a living.
Part of that career was in declarative infrastructure. Infrastructure-as-code rests on the claim that a declaration is the source of truth and not a description of one, and that drift from it is worth building machinery to prevent. A schema that is the lineage is the same claim, moved from infrastructure to scientific data.
Naming the two kinds apart let each be judged on its own terms — the record for how faithfully it is kept, the structure for what it makes impossible. Doug had already made the underlying argument in print, that deterministic provenance has to be a property of the system that produces results, not a record kept alongside it. Naming the two kinds is what made that property easy to see.
“What struck me is how elegant it is. A result can’t exist unless its inputs are there — the provenance falls out of how the pipeline works. I spent a career around records kept next to the data, and that’s what got me: the science comes out more reliable because of how the system is built, not because someone remembered to document it.”
Doug Welsh, Chief Revenue Officer, DataJoint
Both kinds do necessary work. Extrinsic governs a workflow’s boundaries; intrinsic governs what happens inside it. A complete account needs both, joined at interfaces you can point at.
Origin, Contract, Code
DataJoint’s contribution is that the code for a computational transformation is part of the schema. A computed table’s declaration includes the make() that populates it. So classic data integrity comes to subsume computation. The problem provenance was invented to solve, reconnecting a result with the code and inputs that produced it, does not arise as separate bookkeeping, because the data and the transformation were never separated.
Three structural facts follow about every entity: where it came from, what code produced it, and whether it reproduces.
DataJoint marks every table by its origin. Each declaration states where the table’s rows come from, and that marking is the provenance classification. The four names answer one question — what puts rows in this table — which is the question provenance asks. Lookup — the code itself, via contents, so governed by code provenance: which committed version defined it. Entry — something outside the table: a scientist recording a study design, a sample registered on receipt, an instrument’s run sheet. The intrinsic chain begins here, and this is where the extrinsic record has to be captured, because the source is outside what the workflow tracks. Ingest — the table’s own make(), reading an instrument export or an upstream system the workflow does not control: an extrinsic boundary despite being automated. Compute — the table’s own make(), deriving entirely from declared upstream: fully intrinsic and reproducible.
The two nouns are tables somebody else fills. The two verbs are what populate() does. That split is the intrinsic/extrinsic boundary written into the names.
The marking has to match reality. A table filled through a form at run time is an Entry table however it is declared, because code provenance cannot vouch for what a person typed after the code shipped.
Determinism is the make() contract, five rules now published as part of the framework’s documentation (the make() contract, specified in full with rationale and enforcement model in the AutoPopulate reference §4.3):
Populate-only. Rows arrive only through make() via populate(), never a direct insert.
One entity per call, in isolation. One make(key) produces one master row and its parts. No shared state, no ordering, no reading peer rows, which makes populate embarrassingly parallel and each entity’s provenance self-contained.
Read only upstream, restricted to the job. Fetch only declared ancestors reachable from the key. The violation is not “an undeclared table.” It is reading outside the cone.
Write the result to self and its parts, atomically. Writes elsewhere are allowed, provided each destination is given its own extrinsic record.
No other result-affecting input. Arguments to populate orchestrate: batching, parallelism, logging. Anything that changes what is computed enters through a declared upstream table.
A rule you can point at is a rule a reviewer can check, and eventually one a tool can check. As unwritten convention, none of the five was either.
Code provenance is the third pillar, and it is routinely confused with the industry’s software provenance — software bills of materials, supply-chain attestation frameworks such as SLSA, in-toto — which certifies where a software artifact came from and who built it. Code provenance is process provenance in the PROV sense: the code is the activity that produced a data entity, and the record of which version of it ran. The deployment requirement is that populate runs from committed code whose dependencies the repository captures, so that a repo hash is a sufficient representation of what ran. Populating from an uncommitted working tree breaks it. repo hash is a sufficient representation of what ran. Populating from an uncommitted working tree breaks it.
Fan-Out at the Boundary
Fan-out is the established pattern for capturing provenance at the edge of a workflow.
An ingestion table’s make() reads one instrument export — an assay plate read, a chromatography run, a sequencing batch — and, besides its own result, inserts rows into the Entry tables that export describes: the subject, the sample, the session, each carrying the record of where it came from: the source, the agent, the time, and by what means. No foreign key points back to the table that wrote them, and none is needed, because the extrinsic record is the provenance. The contract allows exactly this. A make() may write beyond itself provided it supplies extrinsic provenance to each destination.
The reason the pattern exists is that the workflow changes what it is keyed by at this point. Before ingestion it is keyed by the vehicle that delivered the data: a file, an export, a dataset. After it, entities are extracted and the rest of the workflow is keyed by them: subjects, samples, sessions. The chain leaves the foreign-key graph and re-enters it, so the boundary write is where the account of the outside world gets attached.
The record rides on the destination row rather than in a log beside it, so it cannot be separated from the entity it describes. Only the master carries it; parts inherit it.
The framework produces intrinsic provenance automatically. Supplying extrinsic provenance is the workflow developer’s contract. Even when the source offers little — a nightly pull from a sample-inventory system, say — the author still records what arrived and when. The framework cannot infer what it did not compute.
Where Each Guarantee Is Enforced
The two kinds of provenance are enforced at two different times, and the split follows from what each one depends on.
Intrinsic provenance is settled at schema definition time. The lineage is structural, so whether it holds is a property of the declaration — and a declaration can be checked before it is allowed to define anything. A schema and its transformations are code in a repository: they pass through continuous integration, every branch is read statically, and deployment is pinned to the commit that passed. A transformation that would read outside its declared inputs never deploys, so it never produces a result to audit. This is prevention, and it is available precisely because nothing about it depends on a particular run.
Extrinsic provenance is enforced at run time, because that is when the outside world shows up. What instrument produced this file, which system it came from, who entered it, when it arrived — none of that exists at deploy time, and no static check can supply it. What the deployment can do is refuse to let it go unrecorded: the boundary write carries its source record as the data lands, and a write that arrives without one is not a tolerated gap but a failed ingestion.
So the platform holds both ends. Integrity is guaranteed before anything runs, by admitting only declarations that hold up — the familiar shape of infrastructure-as-code governance, applied to scientific workflows. The account of the outside world is captured as it enters, at the boundary, on the row it describes. Neither can substitute for the other: a static check cannot know what an instrument did last night, and a run-time record cannot make a badly declared workflow consistent.
The stakes rise as agents do the work. An agent produces results faster than anyone can read the record of them, so provenance that depends on being audited falls behind as a matter of arithmetic. What holds at that speed is the guarantee enforced before the run: a result that cannot exist unless its inputs do needs no reader to be true.
Export as Projection
Export is a projection, not a reimplementation. The lineage already exists, complete and consistent; export serializes it into a vocabulary someone else speaks. A row is a PROV Entity, a make() is an Activity, upstream reads are used, and the transitive closure of that upstream trace is wasDerivedFrom. OpenLineage takes the same content in run-centric form, which is how Unity Catalog, Collibra, and DataHub will read it. ALCOA+ maps line by line. Knowledge graphs are the general case, of which a provenance graph is one subgraph.
The mapping is total across the structural core and reaches as far as the deployment records. An auditor verifies the structure itself rather than a record kept beside it.
The data model is the lineage, and it cannot drift from the data, because it is the same thing as the data. That is what it means to sit upstream of the platforms an organization already runs: not another record of the science, but the structure the science is written in.
Integrity first. Provenance follows.
Better science in. Better intelligence out.
References and Figure Credits
Cited in the text:
Codd, E. F. (1970). A relational model of data for large shared data banks. Communications of the ACM, 13(6), 377–387. doi:10.1145/362384.362685
Buneman, P., Khanna, S., & Tan, W.-C. (2001). Why and where: A characterization of data provenance. Database Theory — ICDT 2001, Lecture Notes in Computer Science, 316–330. doi:10.1007/3-540-44503-X_20
Freire, J., Koop, D., Santos, E., & Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in Science & Engineering, 10(3), 11–21. doi:10.1109/MCSE.2008.79
Lim, C., Lu, S., Chebotko, A., & Fotouhi, F. (2010). Prospective and retrospective provenance collection in scientific workflow environments. 2010 IEEE International Conference on Services Computing, 449–456. doi:10.1109/SCC.2010.18
Moreau, L., & Groth, P. (eds.) (2013). PROV-Overview and PROV Primer. W3C Working Group Notes, 30 April 2013.
Yatsenko, D., & Nguyen, T. T. (2026). DataJoint 2.0: A computational substrate for agentic scientific workflows. arXiv. doi:10.48550/arXiv.2602.16585 — the formal definition of the Relational Workflow Model.
OpenLineage specification. LF AI & Data Foundation.
On the term: “intrinsic provenance” is also used, independently and without a shared definition, in artifact authentication — evidence recovered from an artifact’s own material properties rather than from an attached label (Sun, B., Trustworthy provenance for physical and digital artifacts with commodity mobile devices, National University of Singapore, extended abstract). The sense defined above is narrower and belongs to the Relational Workflow Model.
Figures:
Ngram chart — generated from the Google Books Ngram JSON API (en corpus, smoothing 3).
