Where the guarantee comes from

Provenance: A Latecomer

Run “data integrity,” “data consistency,” “data governance,” and “data provenance” through Google’s Ngram corpus, and ask when each of them entered use.

The four terms arrive in order. Integrity comes first, rising from the 1970s — the decade the relational model arrived — and dominating the vocabulary for the next four decades. Consistency follows in the 1980s and settles early at a modest level, where it stays. Governance and provenance don’t launch until the mid-2000s. Then governance climbs steeply enough to very nearly catch integrity by 2022, while provenance lifts only faintly: in the most recent data it still trails integrity by a factor of eight.

The relational model accounts for that gap. Codd and fifty years of database research talk about integrity and almost never about provenance, because the model solved the problem at a deeper level. Provenance became a named discipline in the 2000s in workflow and e-science computing (Buneman 2001; the W3C PROV working group from 2011), where the substrate could not enforce relationships between artifacts. If your results are files produced by scripts, nothing stops you from holding a result whose inputs are gone. So you keep a record beside the data describing what happened, and you work hard to keep that record honest.

Provenance is the complete record of a data entity’s origin: what produced it, from what inputs, under what conditions, and through what chain of transformations.

Customers routinely expect that definition to cover two things it leaves out: the scientist’s reasoning, and governance.

• Provenance records the how, and stops short of the why. The standards ask who, when, and by what means, and so does 21 CFR Part 11. Someone who asks for provenance and means “capture why the scientist chose this analysis” is asking for a different system. Say so early.
• Governance is the wider regime — access, cataloging, audit, retention — and provenance is one component of it. DataJoint produces the authoritative provenance that a governance system consumes.

The literature also splits provenance into prospective, the declared plan, and retrospective, the observed run (Freire 2008; Lim 2010). Integrity is preventive. Provenance tooling is forensic. Prevention and forensics are different guarantees, and much of the confusion in this field comes from buying one and assuming the other.sic. Prevention and forensics are different guarantees, and much of the confusion in this field comes from buying one and assuming the other.

---

The Standards Landscape

Three families of systems account for most of what the field means by provenance tooling. They are also how the rest of the world will phrase its questions.

W3C PROV is the vocabulary: Entities (a thing), Activities (something acting on entities over time), Agents (who bears responsibility), joined by relations like used, wasGeneratedBy, wasDerivedFrom, and wasAttributedTo.

OpenLineage is the same shape recast for data engineering — runs of jobs that read and write datasets — and it is the lingua franca of Airflow, Spark, and Marquez. Governance catalogs — Unity Catalog, Collibra, Alation, Purview, DataHub — record table and column lineage across the enterprise.

Then there is compliance, which is where pharma actually lives. ALCOA+ — attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, available — and 21 CFR Part 11, with audit trails that no one, administrators included, is permitted to delete. ALCOA+ is defined in PIC/S PI 041-1 and in the UK MHRA’s GXP data integrity guidance; Part 11 is 21 CFR Part 11, Electronic Records; Electronic Signatures. These are obligations of a deployment and a process, never features of a library.

In genomics the same shape appears again in the workflow standards of the Global Alliance for Genomics and Health (GA4GH), and in data science in the run-metadata stores that record inputs, parameters, and outputs for each run.

What every one of these families has in common is that the record is kept beside the data. Keeping it there is the only option available when the substrate cannot do better. The consequence is that there are now two things to keep in agreement, and nothing but diligence keeps them there.

---

One Slice Each

Three approaches cover most of what pharma runs today, and each covers its slice well. Document versioning tracks records rather than computation: version every controlled document, with no computational lineage underneath it. That is the quality and regulatory world, where Veeva Vault is the de facto standard. Metadata cataloging renders enterprise lineage as a graph, as Collibra and Informatica do; it traces, it does not enforce. Domain audit logging captures every edit with user, timestamp, and reason inside one boundary, which is what Medidata Rave does for electronic data capture in clinical trials. Large internal integration platforms move research data at enormous scale, with provenance supplied by whatever tooling sits on top of them.

Each covers one slice, well. None makes provenance a property of the data itself. The provenance system and the data system are separate systems, and separate systems can diverge.

---

How the ELN and the Workflow Fit Together

The system a bench scientist actually touches is the electronic lab notebook (ELN): Signals Notebook across most of large pharma, Benchling across much of the adjacent market. It is where the provenance chain starts, and the two systems fit together at that seam. The notebook attests to what was done at the bench. The workflow carries what was computed from it.

Every generation of the notebook, from paper through the AI-assisted tools now arriving, rests on the same mechanism: an experiment is recorded, and a second person reviews it and signs.

That signature is provenance of a specific kind: attestation. A named person asserts that the work was done as described, which is what ALCOA+ codifies and why contemporaneous is on that list. Recorded as the work happens, not reconstructed afterward. It is the right instrument for the bench, where what has to be vouched for is human judgment and physical work. It says someone vouched. It does not say the result could not have been otherwise.

Vince Paton, who spent years selling these systems, draws the line: “An ELN captures the experiment, DataJoint captures the science.” The notebook is authoritative for what a person did at the bench. The workflow is authoritative for what was computed from it. Neither is trying to be the other, and the two kinds of assurance compose: attestation at the boundary, structure downstream of it.

So the integration is a join, not a migration. The notebook’s entry identifiers and sign-off enter the workflow as the extrinsic record on the rows they describe — source, agent, time — and every result computed from those rows carries that identity in its lineage. Ask where a figure came from and the chain runs from the result, through the computation that produced it, back to the signed notebook entry it began in.

The join has to work by reference rather than by copy, and that is the practical requirement customers state themselves. Pointing an ELN’s search at a lakehouse or an object store would mean replicating that data into its backend, which they decline to do. Nothing needs replicating. The ELN keeps its record, the workflow keeps its structure, and the provenance chain crosses the boundary intact.

---

Two Pictures: Assembled vs Declared

Compare what each approach actually produces.

First, from the W3C PROV Primer, the complete provenance graph for one newspaper article: every entity, activity, agent, and relation needed to describe how a single artifact came to be.

It is assembled by observation, it is stored next to the thing it describes, and keeping it accurate is somebody’s job.

Now the schema of a liquid chromatography–mass spectrometry (LC-MS) workflow: twelve entities and their parts, dependencies running left to right, origins carried by shape and color.

That one diagram is the lineage of every row in every table. Nobody assembled it. It was declared before any data existed, and the database refuses to hold a computed result whose upstream dependencies are not present and correct.

A catalog can disagree with the data it describes; a schema can’t.

Why can a catalog drift when a schema cannot? Because the two artifacts in the second picture — the data and its lineage — are not two artifacts. They are the same one.

---

Provenance: Extrinsic vs Intrinsic

This is the distinction the rest of the vocabulary hangs from.

	Extrinsic	Intrinsic
Source of guarantee	Observational: recorded as metadata	Structural: entailed by the data model
Posture	Forensic, after the fact	Preventive, before the fact
Failure mode	Can be incomplete, or drift	Cannot be created broken
Answers	“This datum was reported to come from there”	“This result must derive from these inputs”

As used here the terms are specific to the Relational Workflow Model, the model DataJoint introduces, formally defined in Yatsenko & Nguyen 2026. That is the definition this essay argues from. Intrinsic means entailed by the schema and enforced by referential integrity, not merely inherent in some general sense.

Extrinsic provenance is a record captured about data whose origin lies outside the model’s structural guarantees: read from an instrument, transcribed from a notebook, pulled from a laboratory information management system or a sample registry. Every system in the preceding sections produces this kind, and produces it well. Intrinsic provenance is entailed by the structure of the Relational Workflow Model and maintained by the database itself. It is not a record kept about the data. It is the data model.

Mapped onto the literature: extrinsic corresponds to retrospective provenance, intrinsic to prospective, with one sharpening. In DataJoint the prospective structure is enforced by referential integrity, so it is a binding guarantee rather than a declared intention.

The pair of terms came to DataJoint from Doug Welsh, in February 2026, from the commercial side rather than the engineering one and new enough to still hear the vocabulary the way a customer hears it. Two claims were sharing one word: customers who said they already had provenance had a careful record of what had happened, and DataJoint was offering something that is not a record at all.

The available reading was that DataJoint keeps a better record than the alternatives. He took the harder one: that it keeps no record, and the guarantee comes from somewhere else.

That reading was available to him because he had spent a career selling the alternatives: data virtualization into pharmaceutical accounts, the data lake and its catalog, unified storage, array stores for scientific data. Every one of those systems keeps a record about data it does not control, which is the extrinsic model sold for a living.

Part of that career was in declarative infrastructure. Infrastructure-as-code rests on the claim that a declaration is the source of truth and not a description of one, and that drift from it is worth building machinery to prevent. A schema that is the lineage is the same claim, moved from infrastructure to scientific data.

Naming the two kinds apart let each be judged on its own terms — the record for how faithfully it is kept, the structure for what it makes impossible. Doug had already made the underlying argument in print, that deterministic provenance has to be a property of the system that produces results, not a record kept alongside it. Naming the two kinds is what made that property easy to see.

“What struck me is how elegant it is. A result can’t exist unless its inputs are there — the provenance falls out of how the pipeline works. I spent a career around records kept next to the data, and that’s what got me: the science comes out more reliable because of how the system is built, not because someone remembered to document it.”

Doug Welsh, Chief Revenue Officer, DataJoint

Both kinds do necessary work. Extrinsic governs a workflow’s boundaries; intrinsic governs what happens inside it. A complete account needs both, joined at interfaces you can point at.

---

Origin, Contract, Code

DataJoint’s contribution is that the code for a computational transformation is part of the schema. A computed table’s declaration includes the make() that populates it. So classic data integrity comes to subsume computation. The problem provenance was invented to solve, reconnecting a result with the code and inputs that produced it, does not arise as separate bookkeeping, because the data and the transformation were never separated.

Three structural facts follow about every entity: where it came from, what code produced it, and whether it reproduces.

DataJoint marks every table by its origin. Each declaration states where the table’s rows come from, and that marking is the provenance classification. The four names answer one question — what puts rows in this table — which is the question provenance asks. Lookup — the code itself, via contents, so governed by code provenance: which committed version defined it. Entry — something outside the table: a scientist recording a study design, a sample registered on receipt, an instrument’s run sheet. The intrinsic chain begins here, and this is where the extrinsic record has to be captured, because the source is outside what the workflow tracks. Ingest — the table’s own make(), reading an instrument export or an upstream system the workflow does not control: an extrinsic boundary despite being automated. Compute — the table’s own make(), deriving entirely from declared upstream: fully intrinsic and reproducible.

The two nouns are tables somebody else fills. The two verbs are what populate() does. That split is the intrinsic/extrinsic boundary written into the names.

The marking has to match reality. A table filled through a form at run time is an Entry table however it is declared, because code provenance cannot vouch for what a person typed after the code shipped.

Determinism is the make() contract, five rules now published as part of the framework’s documentation (the make() contract, specified in full with rationale and enforcement model in the AutoPopulate reference §4.3):

1. Populate-only. Rows arrive only through make() via populate(), never a direct insert.
2. One entity per call, in isolation. One make(key) produces one master row and its parts. No shared state, no ordering, no reading peer rows, which makes populate embarrassingly parallel and each entity’s provenance self-contained.
3. Read only upstream, restricted to the job. Fetch only declared ancestors reachable from the key. The violation is not “an undeclared table.” It is reading outside the cone.
4. Write the result to self and its parts, atomically. Writes elsewhere are allowed, provided each destination is given its own extrinsic record.
5. No other result-affecting input. Arguments to populate orchestrate: batching, parallelism, logging. Anything that changes what is computed enters through a declared upstream table.

A rule you can point at is a rule a reviewer can check, and eventually one a tool can check. As unwritten convention, none of the five was either.

Code provenance is the third pillar, and it is routinely confused with the industry’s software provenance — software bills of materials, supply-chain attestation frameworks such as SLSA, in-toto — which certifies where a software artifact came from and who built it. Code provenance is process provenance in the PROV sense: the code is the activity that produced a data entity, and the record of which version of it ran. The deployment requirement is that populate runs from committed code whose dependencies the repository captures, so that a repo hash is a sufficient representation of what ran. Populating from an uncommitted working tree breaks it. repo hash is a sufficient representation of what ran. Populating from an uncommitted working tree breaks it.

---

Fan-Out at the Boundary

Fan-out is the established pattern for capturing provenance at the edge of a workflow.

An ingestion table’s make() reads one instrument export — an assay plate read, a chromatography run, a sequencing batch — and, besides its own result, inserts rows into the Entry tables that export describes: the subject, the sample, the session, each carrying the record of where it came from: the source, the agent, the time, and by what means. No foreign key points back to the table that wrote them, and none is needed, because the extrinsic record is the provenance. The contract allows exactly this. A make() may write beyond itself provided it supplies extrinsic provenance to each destination.

The reason the pattern exists is that the workflow changes what it is keyed by at this point. Before ingestion it is keyed by the vehicle that delivered the data: a file, an export, a dataset. After it, entities are extracted and the rest of the workflow is keyed by them: subjects, samples, sessions. The chain leaves the foreign-key graph and re-enters it, so the boundary write is where the account of the outside world gets attached.

The record rides on the destination row rather than in a log beside it, so it cannot be separated from the entity it describes. Only the master carries it; parts inherit it.

The framework produces intrinsic provenance automatically. Supplying extrinsic provenance is the workflow developer’s contract. Even when the source offers little — a nightly pull from a sample-inventory system, say — the author still records what arrived and when. The framework cannot infer what it did not compute.

---

Where Each Guarantee Is Enforced

The two kinds of provenance are enforced at two different times, and the split follows from what each one depends on.

Intrinsic provenance is settled at schema definition time. The lineage is structural, so whether it holds is a property of the declaration — and a declaration can be checked before it is allowed to define anything. A schema and its transformations are code in a repository: they pass through continuous integration, every branch is read statically, and deployment is pinned to the commit that passed. A transformation that would read outside its declared inputs never deploys, so it never produces a result to audit. This is prevention, and it is available precisely because nothing about it depends on a particular run.

Extrinsic provenance is enforced at run time, because that is when the outside world shows up. What instrument produced this file, which system it came from, who entered it, when it arrived — none of that exists at deploy time, and no static check can supply it. What the deployment can do is refuse to let it go unrecorded: the boundary write carries its source record as the data lands, and a write that arrives without one is not a tolerated gap but a failed ingestion.

So the platform holds both ends. Integrity is guaranteed before anything runs, by admitting only declarations that hold up — the familiar shape of infrastructure-as-code governance, applied to scientific workflows. The account of the outside world is captured as it enters, at the boundary, on the row it describes. Neither can substitute for the other: a static check cannot know what an instrument did last night, and a run-time record cannot make a badly declared workflow consistent.

The stakes rise as agents do the work. An agent produces results faster than anyone can read the record of them, so provenance that depends on being audited falls behind as a matter of arithmetic. What holds at that speed is the guarantee enforced before the run: a result that cannot exist unless its inputs do needs no reader to be true.

---

Export as Projection

Export is a projection, not a reimplementation. The lineage already exists, complete and consistent; export serializes it into a vocabulary someone else speaks. A row is a PROV Entity, a make() is an Activity, upstream reads are used, and the transitive closure of that upstream trace is wasDerivedFrom. OpenLineage takes the same content in run-centric form, which is how Unity Catalog, Collibra, and DataHub will read it. ALCOA+ maps line by line. Knowledge graphs are the general case, of which a provenance graph is one subgraph.

The mapping is total across the structural core and reaches as far as the deployment records. An auditor verifies the structure itself rather than a record kept beside it.

The data model is the lineage, and it cannot drift from the data, because it is the same thing as the data. That is what it means to sit upstream of the platforms an organization already runs: not another record of the science, but the structure the science is written in.

Integrity first. Provenance follows.

Better science in. Better intelligence out.

---

References and Figure Credits

Cited in the text:

• Codd, E. F. (1970). A relational model of data for large shared data banks. Communications of the ACM, 13(6), 377–387. doi:10.1145/362384.362685
• Buneman, P., Khanna, S., & Tan, W.-C. (2001). Why and where: A characterization of data provenance. Database Theory — ICDT 2001, Lecture Notes in Computer Science, 316–330. doi:10.1007/3-540-44503-X_20
• Freire, J., Koop, D., Santos, E., & Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in Science & Engineering, 10(3), 11–21. doi:10.1109/MCSE.2008.79
• Lim, C., Lu, S., Chebotko, A., & Fotouhi, F. (2010). Prospective and retrospective provenance collection in scientific workflow environments. 2010 IEEE International Conference on Services Computing, 449–456. doi:10.1109/SCC.2010.18
• Moreau, L., & Groth, P. (eds.) (2013). PROV-Overview and PROV Primer. W3C Working Group Notes, 30 April 2013.
• Yatsenko, D., & Nguyen, T. T. (2026). DataJoint 2.0: A computational substrate for agentic scientific workflows. arXiv. doi:10.48550/arXiv.2602.16585 — the formal definition of the Relational Workflow Model.
• OpenLineage specification. LF AI & Data Foundation.
• PIC/S (2021). Good Practices for Data Management and Integrity in Regulated GMP/GDP Environments, PI 041-1 — the definition of ALCOA+ and its nine attributes.
• MHRA (2018). ‘GXP’ Data Integrity Guidance and Definitions, rev. 1 — the ALCOA principles as a regulator applies them.
• FDA. 21 CFR Part 11 — Electronic Records; Electronic Signatures, Title 21, Chapter I, Subchapter A.
• On the term: “intrinsic provenance” is also used, independently and without a shared definition, in artifact authentication — evidence recovered from an artifact’s own material properties rather than from an attached label (Sun, B., Trustworthy provenance for physical and digital artifacts with commodity mobile devices, National University of Singapore, extended abstract). The sense defined above is narrower and belongs to the Relational Workflow Model.

Figures:

• Ngram chart — generated from the Google Books Ngram JSON API (en corpus, smoothing 3).
• PROV core types and the complete example graph — from the W3C PROV Primer, reproduced unmodified under the W3C Document License.
• LC-MS schema — DataJoint diagram output for the lcms-demo schema.
• Fan-out pattern — drawn to the DataJoint diagram notation described in Read diagrams.

In DataJoint, the schema is the workflow. Not a description of the data after it lands, but a formal, executable specification of the process that produces it: tables are the steps, foreign keys are the dependency graph, and each derived table carries the code that computes it. That specification has a name — the Relational Workflow Model — and it answers the question infrastructure architects put to every system they are asked to trust: what is the formal language underneath? What follows is the case for it.

A new class of scientific harnesses arrived in 2025 and 2026. Systems like Edison Scientific’s Kosmos and Anthropic’s Claude Science give AI agents the tools, memory, and guardrails to run long, auditable research campaigns. Each of them had to build or assume a data substrate to stand on. Each drew its boundary just above the researcher’s own experimental data. That is where DataJoint sits: upstream, in the governed foundation where the lab’s own data and workflows live — where scientists and their agents reason over the same model of the science, and where reproducibility, provenance, and trust are properties of the foundation rather than artifacts assembled after the fact. Which raises the question this piece takes up: what is the formal language for the workflows at the heart of that foundation?

The question

What formal language describes a research workflow — the entities, the dependencies among them, and the chain of transformations that turns raw experimental output into a scientific result?

Twenty-eight minutes into a recent conversation with a senior pharma infrastructure architect, he stopped and asked it directly: “What language do we use to describe the process?”

He listed the comparators he had in mind:

• YAML, for network configuration
• Terraform, for cloud infrastructure
• Helm, for Kubernetes packaging
• Ansible, for config-drift management
• Cedar, Amazon’s policy language, for authorization rules

Each is declarative. Each is formal enough that tools can parse it, reason over it, and exchange it. Each became a small standard in its corner of the stack because someone wrote down the spec, and others adopted it.

“If you can define the language or find one that you like,” he said, “that becomes a way to pull it up. I could feed that into anything. I could feed that into an LLM. Now I have written the control plane for where I’m going to use RAG and where I’m going to use context.”

The instinct underneath the question is correct, and it is the same instinct that produces every generation of durable infrastructure: a formal language is what survives the next platform shift. The tools come and go. The spec is what gets standardized, interchanged, and built upon.

So he asked: what is the formal spec for the scientific work DataJoint is recording?

One natural answer is “it’s codified in Python,” and it is a good one: it points to where the schema actually gets written. His question was aiming a level deeper, though. “Python is not a formal spec,” he added. “That’s just a programming language.” Exactly right, and it sharpens the question to its real form: what is the formal spec that Python is a binding to?

The answer to that sharper question was already written down in the DataJoint 2.0 paper (Yatsenko & Nguyen 2026).

---

What “schema” has meant

DataJoint’s core innovation is that the schema is the workflow: the proven principles of relational database design, refined and augmented to make computation first-class in the data model. To see what that adds, it helps to recall what a schema has meant until now.

The word schema is a database term of art. In Edgar F. Codd’s 1970 relational model, a schema is the formal description of stored data: typed columns, primary keys, foreign keys, integrity constraints. Peter Chen’s 1976 entity-relationship model added a conceptual model above that — entities, attributes, and the relationships between them, expressed at the modeling tier before the table layout is fixed. Both formalisms specify what the data looks like once it has landed. Neither specifies how it got there.

The transformation that produced the data has classically lived elsewhere — in application code, ETL scripts, workflow files, ingestion services. The schema describes the resting state; the production process is somewhere else, recorded (if at all) in logs and filenames after the fact.

DataJoint’s revision is one declarative move. The transformation is part of the schema. A make() method attached to each derived (Computed or Imported) table specifies how an entity in that table gets produced — invoked once per tuple in the join of the upstream tables referenced by its foreign keys. The transformation carries the same formal weight as the type of the column it produces. The schema is no longer a description of what exists. It is a specification of how a result comes into being.

This is the move that lets a schema serve as a workflow grammar — not just a data grammar. And that is the strength of a grammar: a small set of rules generates an unbounded space of valid workflows, comprehensible to anyone who knows the rule.

---

The schema is the pipeline

The contrast is easiest to see in a picture: the same liquid-chromatography–mass-spectrometry (LC-MS) pipeline as a traditional relational database would model it, and then as a DataJoint schema.

A traditional database: the same pipeline as a relational schema (crow’s-foot notation). The entities as tables, each relationship named and carrying its crow’s-foot cardinality — the shape only, attributes left off for clarity (from the lcms-demo schema):

A computational database: the same pipeline as a DataJoint schema (equivalent to Figure 1 of Yatsenko & Nguyen 2026):

A traditional database records what happened. A computational database defines how it happens.

The ERD is accurate, and it is static. It shows the entities and how they relate — the shape of the data once it has landed — and nothing about how any of it came to be. It cannot say that ACQUISITION, MASS_ANALYSIS, and PEAK_DETECTION are produced by a computation rather than entered by hand.

The DataJoint schema is the same pipeline. It has the same entities and the same keys. Here, though, the schema specifies both: the data that exists, and the dependencies and code that bring it into existence. Tables are workflow steps. Foreign keys prescribe execution order. Every computed table declares its transformation in its own definition, versioned with your code. So the workflow is self-executing: change an input, and everything downstream recomputes, with full lineage preserved.

One formal system — the Relational Workflow Model — where structure, computation, and integrity are queryable and enforceable together. Reproducible by construction. Reusable by default.

A traditional database	A computational database (DataJoint)
Static — records relationships	Self-executing — manages data and runs computation
Describes the resting state	Specifies how each result is produced
The diagram is drawn about the data	The diagram is generated from the schema. It cannot drift from the pipeline that runs

---

The whole table in four lines

The whole-pipeline picture shows the shape. Zoom in on a single table and the same economy appears in its declaration. Take PeakDetection from the lcms-demo pipeline — the computed table where peaks are found in each spectrum under a chosen parameter set.

In DataJoint, the table definition is the whole spec, in four lines:

Source: src/lcms_demo/pipeline/scan.py; its parents MassAnalysis and PeakDetectionParams live in the same module.

Here is essentially what that compiles to in SQL:

Two declarative lines — -> MassAnalysis and -> PeakDetectionParams — carry what the SQL spends most of its length spelling out: four inherited key columns, a four-part composite primary key, and two foreign-key constraints with their exact column tuples and referential actions. DataJoint says the same thing with far less to read. And it says the one thing the DDL cannot. Because PeakDetection is a computed table, its rows are produced by a make() method attached to this same definition, not entered by hand, and DDL has nowhere to put that. Attaching the transformation to the table itself is the Relational Workflow Model move formalized in DataJoint 2.0 — the subject of the next section.

---

Where the clarity comes from

The DataJoint form is shorter, but the point is not brevity. Each construct carries meaning the SQL has to rebuild by convention:

• A foreign key carries the key, not just a pointer. -> MassAnalysis inherits MassAnalysis‘s entire primary key — subject_id, sample_id, session_idx — in a single line, as both the reference and part of this table’s own identity. You never retype an inherited column. The SQL names all four twice, once in the column list and again inside a constraint. A Computed table’s key is inherited from its dependencies in full.
  
• The --- divider makes identity explicit. Everything above it is what identifies a detection run: which spectra, under which parameters. Everything below is dependent data (total_peaks). SQL relocates that distinction into a trailing PRIMARY KEY (...) clause read separately from the columns it governs.
• A name means one thing, and it is enforced. Because keys propagate by name, subject_id refers to the same subject everywhere it appears. DataJoint’s semantic matching rejects any join between namesake attributes that do not share lineage. SQL’s natural join has no such guard: two unrelated id columns match silently.
  
• The dependency is the execution order. -> MassAnalysis does not only add a constraint. It places PeakDetection downstream of MassAnalysis in the DAG, and dj.Computed together with its make() method declares that the table is produced from its parents. DDL describes storage. It says nothing about what fills the rows, or when.
  
• You declare intent, not mechanism. The DataJoint definition states what the table is: a computed entity, identified by its two parents, holding a peak count. The SQL states how to store it: columns, types, keys, cascade rules. That gap, between a specification and its compiled form, is the same one this essay draws between the Relational Workflow Model and the Python that binds it.

---

The answer

The formal definition of scientific workflows — the typed entities, dependencies, and chains of data transformation that produce scientific results, in the same sense that Terraform is the formal specification of desired infrastructure state — is the Relational Workflow Model. It is the central contribution of DataJoint 2.0 (Yatsenko & Nguyen 2026) and it is independent of any particular binding, including the Python one most users see.

A Relational Workflow Model is a declarative description of five things:

1. The entities involved in the work: each a typed relation with an explicit primary key.
   
2. The dependencies between them: foreign keys forming a directed acyclic graph.
   
3. The transformation that produces each derived entity: a make() method on each Computed or Imported table, declaring how each tuple in the join of the upstream foreign-key parents becomes a downstream tuple.
   
4. The integrity constraints that must hold: referential, type, and uniqueness constraints, all enforced by the database itself rather than by convention.
   
5. The state of every derivation: Computed and Imported tables record, for every entity that should exist, whether it does, when it was produced, from what, and by whom.

That is a formal specification. Machines can parse it, tools can reason over it, and it is complete: from the model alone one can recover not only what the workflow is supposed to produce, but the rules for the transformations that produce it, the constraints they must respect, and the state of having (or not having) produced it.

Python is the binding. The binding is expressive and convenient, but it is not the spec. The spec lives one level below the binding, in the structure of the model itself. The grammar, semantics, query algebra, and machine-readable surface are written up in the new Schema as a Workflow Specification concept page.

In a follow-up conversation a few days later, the same architect saw the model demonstrated in a live workflow — every derivation row carrying the git hash of the code that produced it, the foreign-key DAG visible at a glance, every recomputation kicked off by deleting a row and letting the engine bring the world back into agreement with the schema. His phrase for what he was looking at: “You can now go build a dependency tree of provenance.” That is what the Relational Workflow Model produces, by construction. The model is not a new artifact. It is the formalization of what infrastructure architects already build for every other long-lived system, made explicit for the workflows that produce scientific results.

---

What “formal” gets us that the comparators do not

The architect’s instinct was that a formal language is what makes ecosystem interoperability possible. The question is whether scientific work fits in any of the existing languages.

It does not — at least, not without losing what makes the work scientific.

YAML, Terraform, Helm. These describe a desired static state: this is the network I want, this is the service I want to deploy, this is the cluster I want to stand up. They do not describe derivation. The derivation is delegated to an engine outside the spec — Terraform’s planner, Helm’s templating, Ansible’s runtime. That works for infrastructure because the desired state is the contract. For scientific work, the derivation is the contract. Outsourcing it to an engine breaks the spec at exactly the point where the spec needs to be authoritative.

Cedar. Cedar describes authorization policy: who may do what under what conditions. It is a fine formal language for one slice of a control plane: the gate-keeping slice. It does not describe the process the gates are gating.

CWL, Nextflow, Snakemake. These come closer. They were designed to describe scientific workflows. They do not, however, attach the transformation to the dependency edge as a single construct, and they do not unify data state with workflow state. The workflow lives in one file. The data lives somewhere else. The integrity of the join between them is the user’s responsibility. The structural side-by-side runs across nine concerns: data structure, types, foreign-key integrity, computation, execution order, provenance, drift detection, query interface, and retry. That comparison is laid out in the Comparison to Workflow Languages docs page.

We have tested this directly. A compiler we built takes a CWL project — using a 13-step GATK whole-genome-sequencing pipeline with four levels of nested sub-workflows as the worked example — and produces a self-contained DataJoint workflow: dj.Computed tables, explicit make() methods, and a foreign-key chain that mirrors the CWL step DAG. The same model also exports to Nextflow DSL2, one process per table, channel wiring mirroring the foreign-key chain. That is the demonstration that no information is locked in. The translation is asymmetric in exactly one place. DataJoint requires what CWL does not specify: entity names, typed primary keys, gather group keys. A short annotation YAML supplies that data structure. In operational terms, the Relational Workflow Model is a strict superset of the workflow-language category: anything CWL describes, the Relational Workflow Model describes — plus the data structure the science actually depends on.

The Relational Workflow Model satisfies a stricter brief than any of these. It is declarative (schema, not script); formal (relational algebra over a typed DAG); executable as its own spec (the spec runs itself — make() lives on each derived table, applied across the join of its upstream parents); code-attached at every dependency (no separate workflow file); integrity-bearing (constraints are part of the spec); and state-aware (the spec includes the current production status of every derivation). No comparator delivers all six.

---

What the formal language unlocks

A formal language is the precondition for interoperability. If the Relational Workflow Model is the spec for scientific workflows — the entities, dependencies, and chains of data transformation that produce scientific results — then:

• An LLM can read it and answer questions about the work without first inferring what was done. The spec is the description.
  
• A Unity-Catalog-style lineage view can attach to it and gain the transformation on each edge. The catalog records column-to-column dependencies; the Relational Workflow Model records what happened between them.
  
• A simulation environment can run forward against it: change one node, propagate the effect, see what breaks before the change is committed. (Network operators have been doing this against network topologies for two decades. Scientific work deserves the same affordance.)
  
• A policy engine like Cedar can attach to specific edges, gating consequential actions. The Relational Workflow Model says what the work is; Cedar says who is allowed to do it. They compose.
  
• A new tool entering the ecosystem can speak the same language as the tools already there. That is the position the architect was pointing at — “that’s why Cedar took off — Amazon said, this is a really useful way to get people to come play our policy game.”
  

The Relational Workflow Model is the version of that play for scientific workflows. It is already specified. What remains is to set the model on the table next to Terraform and Cedar, so that the people who ask the formal-language question recognize the answer when they see it.

---

What adoption looks like

The architect’s harder question was the second one: how does a formal language actually become a standard? If interoperability is the prize, it is unclaimed until the language is adopted by an ecosystem of tools that did not originally know about it.

His own analogy was Network File System. NFS shipped as a simple, idempotent protocol in 1984. It was, in its first years, dismissed — files over a network, too slow, unreliable. Forty years later it is still in use. NetApp built a multibillion-dollar business on it, and a long tail of storage vendors followed. The protocol won not because it was clever, but because it was small enough to be implementable by anyone who needed it.

The Relational Workflow Model has the same shape. The construction is small: typed entities, an FK-directed acyclic graph, a transformation attached to each edge, a small set of integrity constraints, and the state of every derivation. The DataJoint 2.0 paper (Yatsenko & Nguyen 2026) specifies all five. The grammar is implementable in any host language with a relational substrate. Python is the binding we maintain, but it is not the only possible binding. The path to adoption is the same as NFS: make the spec simple enough that an instrument capture system, a regulatory submission packager, and a literature-evidence engine can each publish into or read from the model without rewriting their own data structures.

That is the work the formal language unlocks. The spec lives one level below the binding. The ecosystem can live one level above.

---

Closing

It was exactly the right question to ask: the kind worth answering with care, in writing, so the answer can be cited rather than restated each time.

The question was: what formal language describes a research workflow — the process that produces a scientific result?

The answer: the Relational Workflow Model. The schema is the specification. Tables are the typed entities, foreign keys are the dependency graph, make() methods on derived tables are the transformations, and integrity constraints and derivation state are part of the spec. Python is one binding; the spec does not depend on it. The model is defined in the DataJoint 2.0 paper (Yatsenko & Nguyen 2026); its grammar, semantics, and algebra are written up in the Schema as a Workflow Specification concept page; and its position among workflow languages is in the Comparison to Workflow Languages page.

This is also the exact meaning of the verb at the foundation of DataJoint’s capability roadmap: codify. To codify an experiment is to express it as a Relational Workflow Model — a formally defined, self-executing workflow whose integrity, provenance, and reproducibility hold by construction and stay free to evolve, so that people and their AI agents build on work that compounds instead of disappearing.

Nothing had to be invented to give this answer. The model has been built into DataJoint’s foundation from the start. Every DataJoint schema is already this specification. What the 2.0 paper (Yatsenko & Nguyen 2026) contributed was to articulate it succinctly and precisely: define the model, name it, and set it beside the other formal languages, so that the people who think in specifications can cite it, compare it, and build on it.

Better science in. Better intelligence out.

A 1-Minute Overview of Neuropixels

What if you could listen to hundreds of brain cells at once and still keep the data trustworthy and easy to share? This is what Neuropixels makes possible and how the community turns those signals into reliable results.

Quick hits: what Neuropixels make possible

Neuropixels are silicon probes that continuously detect tiny voltage changes from nearby neurons. Neuropixels probes, first released in 2017, set a new gold standard for large-scale electrophysiology [1]. The goal was simple to state and hard to do: record many single neurons in freely behaving animals with light, flexible hardware and many recording channels. Earlier probe stacks, such as passive NeuroNexus shanks with INTAN headstages, offered only tens of channels and relied on long analog cables that picked up electrical noise. 

Neuropixels take a different path. Using complementary metal oxide semiconductor (CMOS) fabrication, the probe integrates amplification, multiplexing, and digitization on the device itself. Placing the electronics next to the recording sites shortens analog paths, reduces noise, and enables hundreds of addressable channels. This was not practical earlier because fabrication yields on long  slender shanks were low, power and heat budgets were tight, and packaging had to remain biocompatible. Process advances and coordinated investment made it feasible. The result was a step change in experimental capability: stable, simultaneous recordings from hundreds of neurons across multiple brain regions in freely moving animals [1]. 

Adoption is broad. Our conservative estimate is that more than 850 laboratories have used Neuropixels since 2017*. 

Who built it

The Neuropixels project is an international collaboration led by IMEC (Interuniversity Microelectronics Centre, Belgium) with partners including the Wellcome Trust, Howard Hughes Medical Institute (HHMI) Janelia Research Campus (Timothy Harris and colleagues), University College London (Matteo Carandini, Nicholas Steinmetz, and collaborators), the Allen Institute for Brain Science (Christof Koch and colleagues), Gatsby Charitable Foundation, the Sainsbury Wellcome Centre (John O’Keefe and colleagues), among others acknowledged in the original publications [1-3]. Their collaboration transformed what was once a technological aspiration into an accessible community resource. Neuropixels 2.0, co-developed with Cambridge NeuroTech, built upon this foundation with improved reusability and chronic stability [4,5].

From 1.0 to Opto: The Probe Line-Up

The development of Neuropixels has progressed through a series of significant releases, each enhancing its capabilities and resolution. This evolution highlights the ongoing advancements in neural recording technology. 

Most Neuropixels models allow you to select up to 384 channels to record simultaneously from many more physical sites on the shank. The variants below differ mainly in site geometry, density, shank layout, and added capabilities (primate length, ultra-dense mapping, or built-in light for optogenetics).

Since its debut in 2017, the Neuropixels series has revolutionized neuroscience research. The original Neuropixels 1.0 featured 960 recording sites setting a new standard for high-density neural data collection in rodents. Building on that success, Neuropixels 2.0 arrived in 2021 with 1,280 recording sites per shank, ideal for chronic experiments with freely moving animals. By 2023, the lineup expanded further to include Neuropixels NHP for primates and clinical research. The same year, Neuropixels Ultra dramatically increased spatial resolution by pushing site density even higher for more precise spike detection and cell-type identification. Most exciting, the 2025 introduction of Neuropixels Opto combines electrophysiology with neurophotonics, enabling simultaneous electrical recordings and optical manipulation of neuronal activity through optogenetics.

These innovations represent a significant leap forward in neural research, evolving from small-scale rodent studies to detailed, large-scale analyses of the brain system. This includes ultra-high-resolution mapping and multimodal data integration, opening an exciting new chapter in neural exploration.

The Three-Step Guide to Choosing a Probe 

Different probe designs serve different needs. Your choice should follow the question, species, surgery plan, recording duration, and budget.

Neuropixels integrates on-probe CMOS electronics with very high site counts per shank. Helpful for dense, stable, multi-region recordings in freely moving animals. 

NeuroNexus offers a broad range of passive silicon probes in many geometries for acute and chronic in vivo recordings. Probes pair with external or headstage electronics, which keep them slender and flexible in configuration. Compared with Neuropixels, per-shank site counts are lower, with 32 sites per shank, and digitization occurs off-probe. Useful when you need custom layouts or thinner shanks.

INTAN Technologies offers low-noise amplification and digitization on the headstage used across many passive probes, including NeuroNexus and Cambridge NeuroTech. Adds headstage weight and analog cable length compared with on-probe designs. Flexible and cost-aware. 

Cambridge NeuroTech focuses on chronic-focused silicon passive probes and implant hardware; partner on Neuropixels 2.0. Pairs with external electronics; their strengths include chronic implant hardware and diverse probe geometries. Strong for long-term stability and surgical accessories.

NeuroSeeker (EU-funded project) was an earlier initiative involving the Harris Lab at Janelia, IMEC, and UCL that advanced dense CMOS concepts that informed today’s designs. 

The Masmanidis Lab at UCLA creates silicon probes with custom geometries, often distributed through collaborations with NeuroNexus or Cambridge NeuroTech. Good for specialised layouts.

Diagnostic Biochips builds high-density, small-form-factor probes designed for chronic rodent work. Emphasizes compact headstages and implant practicality.

How to choose:

1. Need highest per-shank site density and fewer external cables? Consider on-probe-CMOS designs.
2. Need unusual geometries or ultra-thin shanks? Consider passive silicon lines with external headstages.
3. Long-term chronic implant with specific surgical hardware? Choose vendors with mature chronic ecosystems (e.g., Cambridge NeuroTech, Diagnostic Biochips) and proven accessories.

From Acquisition to Sorting: The Open Toolchain at a Glance

Open tools made Neuropixels practical at scale:

• SpikeGLX for high-performance data acquisition software optimized for Neuropixels. It enables stable, low-latency streaming of hundreds of channels. Developed by Bill Karsh (HHMI Janelia) and colleagues.
• Open Ephys for a modular, open-source acquisition hardware and software for real-time electrophysiology. While compatible with multiple probes, it has been widely adopted for Neuropixels acquisition. Created by Josh Siegle (Allen Institute) and Jakob Voigts (MIT, Open Ephys company). 
• Kilosort for fast, GPU-based spike sorting tuned for dense data produced by Neuropixels. Developed by Marius Pachitariu and collaborators (originally at UCL and Janelia).
• SpikeInterface for standardized preprocessing, spike sorting, validation, and comparison across multiple algorithms, including Kilosort. It integrates smoothly into Neuropixels workflows. Led by Alessio Buccino and Cyrille Rossant with an international community.
• The Allen Institute Ecephys pipeline for quality metrics and alignment practices, forming the backbone of the Allen Brain Observatory’s large-scale projects. Maintained by Josh Siegle and the Allen Institute team. 

Together, these tools create an ecosystem that supports the acquisition, processing, and analysis of Neuropixels data with rigor and reproducibility. Their open-source nature has ensured broad adoption across labs worldwide, cementing Neuropixels as the foundation for modern systems neuroscience.

DataJoint in Action: What You Get

Running Neuropixels at scale is a data complexity problem, especially when done in multimodal experiments synchronizing other instruments.

DataJoint directly addresses the challenge of managing and analyzing terabyte-scale Neuropixels datasets in combination with other instruments as reproducible, cloud-ready workflows. 

Open-Source DataJoint Pipeline at hand

The Element Array Ephys, a component of the NIH U24-funded DataJoint Elements initiative, delivers validated Neuropixels workflows for acquisition and analysis: 

1. Ingests from SpikeGLX and Open Ephys automatically
2. runs spike sorting with Kilosort and other algorithms through SpikeInterface
3. Syncs and integrates with behavior, stimulation, and imaging
4. Writes results to Neurodata Without Borders (NWB) for sharing
5. Stores everything in structured, queryable databases for collaborative work.

Labs apply these approaches to deliver rigorous, reproducible analyses on large datasets. DataJoint helps you spend less time on infrastructure and more time on experiments.

Rigor and Transparency

The DataJoint Platform is a computational environment for scaling modern neuroscience — on the cloud or on premises in research labs. It automates steps from acquisition to sorting and analysis, scales to terabytes, integrates electrophysiology with multiple modalities, including behavior, imaging, and stimulation, and ships as open, community-validated workflows. FAIR practices and NWB export are built in. Additional capabilities include manual spike curation, advanced quality metrics for spikes and units, event-aligned analysis (e.g., PSTHs), and efficient compression of raw data for faster uploads. 

‍

Looking Ahead

Neuropixels continue to expand the boundaries of what is experimentally possible. As density and multimodal designs grow, workflows must keep pace. By working with the community and delivering scalable, cloud-based pipelines, DataJoint helps turn rich recordings into reliable, shareable science. 

Upload. Analyze. Share. No local infrastructure required.

• Explore Element Array Ephys tutorials:  https://github.com/datajoint/element-array-ephys 
• Contact us for a demo of DataJoint SciOps for Neuropixels. ‍
• See you at SfN 2025!  We’re excited to showcase our latest research. Don’t miss our nanosymposium presentation titled “Parametric Stimuli Reveal Functional Subcircuits in Visual Cortex.” Be sure to visit our poster, #PSTR198, where we’ll present “A Principled Framework for Compression and Standardization of Multiphoton Data.” Excited to connect with you there! 

‍

---

* Estimated from PubMed papers mentioning ‘Neuropixels’ since 2017, de-duplicated by institution and counting consortium papers once. This is a conservative lower bound.

References

[1] Jun, J. J., Steinmetz, N. A., Siegle, J. H., et al. (2017). Fully integrated silicon probes for high-density recording of neural activity. Nature, 551(7679), 232–236.

[2] Simons Foundation (2017). ‘Neuropixels’ expand access to the brain

[3] The Brain Probe Consortium: Neuropixels silicon probes.

[4] Steinmetz, N. A., Koch, C., Harris, T. D., and Carandini, M., et al. (2021). Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings. Science.

[5] Siegle, J. H., et al. (2017). Open Ephys: An open-source, plugin-based platform for multichannel electrophysiology. J Neural Eng, 14(4).

‍

‍

(Previously in this series: “Power of Schemas,” which detailed the theoretical foundations of structured data, and “The Great Data Debate,” which introduced schema-on-write vs. schema-on-read.)

---

AI and the Evolution of Relational Schemas

The argument often surfaces that Artificial Intelligence thrives on unstructured data, framing the “rigidity” of schemas (as discussed in our second post) as a hindrance. However, this perceived rigidity is precisely what ensures data integrity—the accuracy, consistency, and reliability of data. And for AI to produce trustworthy results, integrity is paramount.

Why AI Still Needs a Backbone of Integrity

Key aspects of data integrity, often enforced by well-defined schemas, include:

• Entity Integrity: Ensuring each real-world entity is uniquely identified. Think of this as every citizen having a unique ID, preventing confusion.
• Referential Integrity: Guaranteeing that relationships between data remain valid. This ensures, for example, that lab results are correctly linked to the specific patient, preventing critical misattributions.
• Group (Compositional) Integrity: Treating entities composed of multiple essential parts as inseparable units. For instance, if an algorithm extracts several signal traces from one recording, group integrity ensures these are managed as a complete set; removing one arbitrarily would invalidate the analysis.

The rise of AI doesn’t fundamentally change the tradeoffs between schema-on-write and schema-on-read (explored in our first post). While AI can process unstructured “data soup,” an AI working with well-structured data is like a detective with neatly organized evidence logs—connections are clearer, verification is easier, and conclusions are far more reliable. In fact, an AI might even express its understanding of unstructured data by constructing a relational schema, offering a verifiable representation of its inferred findings.

Evolving the Model for Modern Data Challenges

Traditional relational implementations do face challenges with modern data:

1. Handling Large Objects: Efficiently storing and querying massive objects like videos or raw instrument outputs can be impractical in classic relational structures.
2. Schema Evolution: Modifying schemas in large, live databases can be cumbersome, hindering agility.
3. Integrating Computation: Deeply embedding complex computations (often in Python) and managing their dependencies within the data model requires extensions beyond standard relational frameworks.

Addressing these means evolving the relational approach. New models need to support large objects more natively, allow schemas to adapt without sacrificing integrity, and treat computation as an integral part of the data pipeline. AI itself could aid this, potentially inferring relational structures from unstructured data, providing a verifiable hypothesis about its underlying organization.

DataJoint: A Modern Example of Structured, Computable Data Management

The DataJoint framework exemplifies such an evolved, structured approach, especially for scientific AI applications. It refines the relational model by integrating computational dependencies directly into the schema. This treats computations as first-class citizens, allowing entire scientific workflows—from data acquisition and processing to analysis—to be represented as a unified, integrity-checked data pipeline. Imagine a digital lab notebook combined with an automated assistant, where every experiment (computation) is precisely linked to its data inputs and methods, ensuring results are traceable, verifiable, and reproducible.

The Enduring Need for Structure

Ultimately, the choice between structured, unstructured, or hybrid data strategies depends on specific needs. Where rapid ingestion of diverse data is key and some inconsistency is tolerable, schema-on-read holds advantages. However, for systems demanding high data integrity, consistency, and provable relationships—especially when AI is involved in critical decision-making—the mathematical rigor and enforcement capabilities of well-defined schemas remain essential. AI is a powerful analytical tool, but it doesn’t negate the foundational need for structure when trustworthiness and reliability are non-negotiable.

---

This article is Part 3 of our three-part series, Entrepreneurs of Insight. In Part 1, we traced the breakdown of the old compact between science and society. In Part 2, we examined the promise and peril of the Executive Order on “Restoring Gold Standard Science.” Here, we introduce a forward-looking model: Insight Entrepreneurship.

---

Introducing: Insight Entrepreneurship

This new vision, Insight Entrepreneurship, reframes the role of scientists and research entities. It calls for them to become proactive stewards and developers of knowledge, taking greater ownership of their intellectual endeavors and the insights they produce. “Insight” is the core currency – encompassing not only discoveries with commercial potential but also fundamental breakthroughs that answer deep questions and satisfy broad human curiosity. “Entrepreneurship” here signifies a mindset of innovation, strategic resource management, value creation (intellectual, societal, and economic), and accountability for the integrity and impact of one’s work.

Core Tenets of Insight Entrepreneurship:

The Scientist as an Empowered Entrepreneur of Insight: Individual researchers and their teams are viewed as primary engines of discovery and “entrepreneurs of knowledge.” This vision empowers them with significant autonomy and encourages a form of ownership over their core intellectual contributions – including methodologies, experimental designs, software, curated datasets, and the unique insights generated. This stewardship extends to having substantial strategic input into how research funding is deployed and leveraged, enabling them to build and sustain their “intellectual capital” for long-term, impactful inquiry. The goal is to foster an environment where brave, independent, and creative endeavors can thrive.

Accountability through Radical Integrity and Verifiability: This enhanced autonomy and ownership are inextricably bound to a profound and demonstrable commitment to radical integrity. Researchers and their institutions must champion and adhere to the highest standards of rigor, transparency in methodology, and verifiability of findings. While the immediate, unconditional release of all raw data from publicly funded projects may not always be the optimal strategy for building intellectual capital or ensuring data quality and context, verifiability remains paramount. The scientific community must develop and adopt robust mechanisms to ensure that all claims are traceable, methodologies are transparent (even if access to underlying raw datasets is managed), and findings can be independently scrutinized and validated under clear, ethical guidelines. This demonstrable integrity is the bedrock of trustworthiness.

Strategic Management of Knowledge Assets for Diverse Value Creation: All outputs of the research process – from foundational data and sophisticated workflows to transformative insights – are treated as valuable knowledge assets. The scientists and institutions who create these assets must strategically manage them to maximize their diverse forms of value. This includes establishing clear frameworks for knowledge ownership, data governance, responsible licensing, and ethical commercialization where appropriate. The aim is to ensure that insights are not only generated but are also effectively translated into broader intellectual advancements, societal benefits, or economic opportunities, with benefits flowing back to support further research and innovation.

A Diversified, Agile, and Sustainable Funding Ecosystem: While core public funding for foundational (“blue sky”) research remains essential, Insight Entrepreneurship fosters a more diversified and resilient funding landscape. This includes promoting more dynamic and transparent collaborations with industry, encouraging philanthropic investment in bold ideas, and exploring models where the value generated from knowledge assets (e.g., through licensing or spin-offs) can create sustainable revenue streams to support ongoing research, reducing sole reliance on fluctuating government appropriations.

Cultivating Intellectual Freedom and Robust Scientific Discourse: A core goal of Insight Entrepreneurship is to foster an environment resilient to ideological capture and conducive to healthy, rigorous, and open debate on all scientific questions, including those that are socially or politically sensitive. By empowering individual researchers with greater ownership of their intellectual trajectory and demanding accountability based on evidence and integrity rather than conformity, the system becomes more decentralized. This decentralization, coupled with a renewed commitment from the scientific community itself to uphold principles of free inquiry and viewpoint diversity (within the bounds of ethical and evidence-based practice), can provide a stronger defense against the imposition of orthodoxy, “runaway ideologies,” or self-censorship. The “market” for insights, in this broader sense, should ultimately favor those that are most robust, evidence-based, and generative, regardless of their alignment with prevailing dogma.

Principled and Impactful Communication: Research communication must be didactic, aiming to inform and educate both the scientific community and the public with clarity and unimpeachable honesty. A distinction is maintained between the often messy, complex reality of the research process and the focused, verifiable claims made in its communication. In an era of Insight Entrepreneurship, where the perceived value of insights is critical, ensuring that communications are both compelling and scrupulously honest is fundamental to building and maintaining trust.

The Role of Advanced Research Platforms in Enabling Research Entrepreneurship

The vision of Insight Entrepreneurship, with its emphasis on researcher ownership, demonstrable integrity, and the strategic management of complex knowledge assets, necessitates advanced infrastructure. DataJoint emerges as a principal tool and foundational platform designed to empower individual researchers and teams to thrive in this new era. It enables them to meticulously create, preserve, and demonstrate their key capabilities, methods, and outputs under conditions of operational excellence and data integrity.

Specifically, DataJoint facilitates:

• Creation and Preservation of Intellectual Assets: It provides a robust framework for systematically defining, executing, and evolving complex research workflows, treating the logic, the acquired data, and analytical results as interlinked, versioned assets. This inherently documents the research process, securing intellectual property and ensuring clear provenance for every insight.
• Operational Excellence and Data Integrity: Through standardized data models, automated processing pipelines, and support for FAIR (Findable, Accessible, Interoperable, Reusable) principles, DataJoint underpins the operational excellence required for modern, data-intensive science. This systematic approach is fundamental to ensuring the integrity and reliability of the knowledge assets generated.
• Demonstrable Integrity and Verifiability: DataJoint’s architecture enables communicated findings to be directly and transparently traced back to their origins within the managed research pipeline, including specific data versions, analytical methods, and computational steps. This traceability is crucial for upholding the principle of radical integrity and allowing for independent verification under agreed-upon protocols. DataJoint enables research teams to efficiently implement and certify “Gold Standard Science.”
• Strategic Management and Collaboration: It allows for granular control over data and workflow components, enabling researchers and institutions to strategically manage their intellectual capital – sharing elements with global partners to foster collaboration while protecting core proprietary assets. This supports the clear delineation and enforcement of governance and ownership structures vital for both academic and commercial translation of insights.

By providing such comprehensive capabilities, platforms like DataJoint empower insight entrepreneurs to not only generate groundbreaking discoveries but also to manage them as durable, verifiable, and valuable contributions, thereby meeting the highest standards of scientific rigor and public accountability.

The Goal: A Revitalized Scientific Enterprise

Insight Entrepreneurship aims to revitalize the scientific enterprise by fostering a culture rooted in individual initiative, profound integrity, and the strategic management of valuable knowledge. It seeks to build a system that is more agile, more accountable, sustainably funded, and deeply impactful. By empowering scientists as entrepreneurs of insight, responsible for both the creation and stewardship of knowledge assets, this vision endeavors to restore a nuanced public trust, ensure science’s intellectual leadership, and unlock new frontiers of discovery for the benefit of all.

---

This concludes the three-part series, Entrepreneurs of Insight. It begins with Part 1, A New Course for Scientific Discovery.

This article is Part 2 of our three-part series, AI Needs Data Discipline. In Part 1, we explored schema-on-write vs. schema-on-read and how hybrid systems emerged. Here we turn to the mathematical foundations: how relational models, entity-relationship diagrams, and schemas express relationships more powerfully than metadata alone. In Part 3, we’ll examine how these foundations must evolve for modern AI-driven challenges.

---

The structured data approaches, particularly the schema-on-write philosophy we discussed in our previous post, weren’t born out of a desire for corporate rigidity. Their origins are deeply rooted in mathematical rigor and the quest for expressive, provable methods of managing data.

The Mathematical Bedrock of Order

The intellectual lineage of structured data traces back to 19th-century mathematicians like De Morgan, Boole, and Cantor, who formalized logic and set theory. These mathematical tools laid the groundwork for the relational data model, which was formalized by Edgar F. Codd in the late 1960s and early 1970s. Codd’s model was a direct application of Set Theory and predicate logic to data management. Designing schemas was no longer like intuitively nailing boards together; it’s akin to using precise engineering principles—physics, material science, geometry—to design a bridge, ensuring its stability and longevity through provable calculations.

Before the relational model, data systems like hierarchical and network models often embedded relationships directly within data records. While functional, they could be complex to query and lacked a strong theoretical basis for data independence and integrity. Codd’s innovation was to represent data as mathematical “relations” (visualized as tables), where relationships are expressed through shared values (keys) rather than physical pointers. This offered a clear, declarative way to define data structures (schemas), enforce constraints, and query data using logical operations. The goal was precision and consistency, not arbitrary inflexibility.

Building on this, Peter Chen introduced the Entity-Relationship Model (ERM) in 1976. While Codd provided the mathematical underpinnings, Chen’s ERM offered a more intuitive, conceptual way to design databases. ERM focuses on identifying “entities” (e.g., ‘Customers,’ ‘Products’) and the “relationships” between them (e.g., a ‘Customer’ places an ‘Order’). Entity-Relationship Diagrams (ERDs) became a standard graphical tool to visualize these, acting as a blueprint before database implementation. It’s important to note that the relational model and ERM are the foundational principles, while SQL (Structured Query Language) is the common language used to implement these principles in databases.

Metadata Implies Relationships whereas Schemas Express and Enforce Them

So, how do we truly “understand” the relationships within data? One could argue this understanding is crystallized through the act of constructing a schema.

Metadata, or “data about data,” is incredibly valuable. It provides context, aids discoverability, and tracks provenance. For instance, metadata is like tagging a passenger with her destination and her luggage with her name. This provides useful context for her journey.

A formal schema, on the other hand, expresses and enforces these relationships as an intrinsic, verifiable part of the data system that supports an enterprise. Continuing our travel analogy, the schema is what guarantees the passenger her assigned seat on the correct flight and ensures her luggage makes the correct flight transfers. Foreign key constraints within a schema don’t just describe a link; they actively prevent operations that would violate that link, ensuring referential integrity is maintained by the database itself. This active enforcement provides a far stronger guarantee of consistency than descriptive metadata alone.

---

While the relational model provides a powerful foundation, how does it fare against the scale and complexity of modern data, especially with the rise of AI? In Part 3 of AI Needs Data Discipline, “AI and the Evolution of Relational Schemas,” we’ll explore these challenges and why the need for structure persists.

This article is Part 2 of our three-part series, Entrepreneurs of Insight. In Part 1, we explored how the post-WWII compact between science and society has collapsed. Here we examine the Executive Order on “Restoring Gold Standard Science” – its rationale, benefits, and risks. In Part 3, we introduce a new vision: Insight Entrepreneurship.

---

“Restoring Gold Standard Science”

A significant governmental response to the perceived crisis in science arrived with the Executive Order issued on May 23, 2025, “Restoring Gold Standard Science.” Understanding its rationale, approach, and likely impacts is crucial for charting a more effective path forward for the scientific enterprise.

Stated Rationale and Aims of the Executive Order: The EO directly cites a significant fall in public confidence in scientists, a reproducibility crisis acknowledged by researchers themselves, and high-profile data falsifications. It argues that the Federal Government has contributed to this loss of trust, providing examples such as allegedly misleading COVID-19 school guidance, flawed environmental projections (National Marine Fisheries Service), the controversial use of certain climate change scenarios, and the politicization of science through initiatives like Diversity, Equity, and Inclusion (DEI) in science planning under the prior administration. Its stated purpose is to restore a “gold standard” for science, ensuring federally funded research is transparent, rigorous, and impactful, and that Federal decisions are informed by credible, reliable, and impartial scientific evidence. The EO aims to restore scientific integrity policies of a previous administration and thereby rebuild the American people’s faith in the scientific enterprise.

Critique of the “Restoration” Premise and its Vision: A core tenet of the EO is the “restoration” of a supposed former ideal of scientific practice. However, this premise is problematic. The “gold standard” it seeks to reinstate may be an idealized or romanticized view of a past that never quite existed in such a pristine form, or one that is ill-suited to the complexities of 21st-century science. Science’s relationship with society has always been dynamic and often contested. More importantly, the EO, by its very nature as a corrective and restorative measure, does not offer a new, forward-looking vision for science. It is primarily a framework of control and compliance, focused on rectifying perceived past errors through prescribed standards and procedures. While principles like rigor and transparency are vital, the EO’s approach risks defining them too narrowly or instrumentally, potentially leading to a bureaucratic and defensive posture within the scientific community rather than fostering a proactive culture of innovation and intellectual leadership.

The EO’s impact, while aiming for positive outcomes like enhanced reproducibility and transparency, is likely to be mixed and carries significant risks:

Potential Benefits:

If implemented judiciously, a heightened focus on data transparency, clear articulation of uncertainties, and rigorous peer review could address some valid concerns about scientific practice, particularly in regulatory science.

Significant Risks and Limitations:

• Stifling Innovation: The prescriptive nature of “Gold Standard Science,” with its detailed mandates, could lead to a compliance-driven research environment that discourages novel, high-risk, or unconventional avenues of inquiry – the very approaches that often lead to breakthroughs.
• Politicization of Science Policy: The EO itself is a political document, explicitly reversing policies of a “prior Administration” and targeting specific scientific examples through a particular lens. This sets a precedent for science policy to oscillate with changing administrations, undermining the stable, long-term frameworks necessary for scientific progress. The mechanisms for defining and enforcing “impartiality” or identifying “highly unlikely assumptions” could themselves be wielded for political ends, subtly directing research away from disfavored topics (e.g., certain aspects of climate science or public health).
• Increased Bureaucracy: Implementing and monitoring compliance with the EO’s detailed requirements across all agencies could create significant administrative burdens, diverting resources and time from research itself.
• Focus on Process Over Outcome: An overemphasis on adherence to prescribed processes might not, by itself, guarantee better scientific outcomes or restore deep public trust, which is also built on demonstrated societal benefit and engagement.
• Narrowing of Scientific Inquiry: The EO’s critique of certain scenarios or its discouragement of DEI considerations could inadvertently narrow the scope of scientific inquiry or discourage research into complex societal challenges where such factors are relevant.

Underlying Social and Scientific Trends: The emergence of such an Executive Order is not an isolated event but is underpinned by broader social and internal scientific trends. These include a documented decline in public trust in many institutions (including science and academia), heightened political polarization where scientific findings are often weaponized, and the rapid spread of misinformation that can erode the authority of scientific consensus. Internally, the scientific enterprise has faced valid criticisms regarding reproducibility, transparency, and, as discussed previously, instances where academic communities have struggled to maintain healthy, open debate on contentious issues, sometimes appearing insular or ideologically uniform to segments of the public. These internal vulnerabilities and the external climate of skepticism can create an environment where directive interventions like the 2025 EO are seen by some as necessary or politically opportune.

This Executive Order, therefore, represents a critical juncture. While it responds to some legitimate concerns, its top-down, regulatory, and potentially politicized approach is unlikely to provide the adaptive, resilient, and genuinely empowering framework that science needs to thrive and effectively serve society in the future. This underscores the urgent need for a more fundamental rethinking of the scientific enterprise, driven from within the community itself.

---

‍Continue reading in Part 3 of the series, Insight Entrepreneurship – A New Vision for Science.

This article is Part 1 of a three-part series, Entrepreneurs of Insight. In this opening installment, we examine how the old compact between science and society has broken down. In Part 2, we look at the promise and peril of the recent Executive Order on Restoring Gold Standard Science.

---

A New Course for Scientific Discovery

The historic compact that defined the post-World War II scientific era – characterized by broad public trust in a largely autonomous academic enterprise sustained by generous, relatively unfettered public support – has demonstrably run its course.

Shifting social trends, evolving public expectations, questions regarding the reliability and utility of some academic outputs, and the sheer scale and expense of modern scientific challenges have eroded this old consensus. Science’s principal role is not merely to respond to societal requests or fulfill pre-defined national priorities, but to proactively generate new insights, chart new intellectual directions, and fundamentally expand the horizons of understanding. Simultaneously, the prospect of replacing the old model with new forms of restrictive control, potentially prioritizing narrow agendas over foundational inquiry, threatens to stifle this very creativity, independence, and intellectual leadership that allow science to flourish and genuinely serve society.

Neither a return to an idealized past nor the imposition of overly restrictive external controls offers a viable path forward. A new vision is required – one that empowers the scientific community to proactively address these challenges, rebuild trust through demonstrable integrity and value creation, and ensure science remains a dynamic engine of human understanding and progress.

---

Continue reading in Part 2 of the series, Restoring Gold Standard Science.