DataJoint has achieved Built-On status with DataBricks.

  • When Scientific AI Forgets How It Got There

    Pharma R&D has spent the last three years quietly making AI load-bearing. Models now propose targets, score compounds, design assays, summarize literature, and increasingly draft sections of regulatory documents. The conversation has been about capability — what AI can do. The conversation we are not having, and need to, is about provenance — what the institution can prove about how it did it.

    The argument is narrow and specific: scientific AI without deterministic provenance is not just messy. It is institutionally dangerous.

    By deterministic provenance, I mean the ability to point at any AI-derived result — a hit list, a predicted structure, an annotated cohort, a generated experimental protocol — and reconstruct, bit-for-bit, the data, code, model version, parameters, random seeds, and upstream transformations that produced it. Not “logged somewhere.” Not “the analyst remembers.” Reproducible on demand, by someone other than the person who first ran it.

    Most pharma AI today does not meet this bar. Notebooks reference data files whose schemas have since drifted. Foundation models are version-pinned on paper but rarely in practice. Pipelines stitch together SaaS APIs, internal scripts, and curated spreadsheets. The output looks crisp; the lineage is a smear.

    This is fine when AI is a brainstorming partner. It becomes dangerous the moment AI output enters the institutional record — IND-enabling packages, IP filings, partnership data rooms, internal go/no-go decisions. Three failure modes follow.

    First, regulatory exposure. FDA and EMA expectations are converging on reconstructable analyses. “Our model said so” is not a defense if the model, weights, and inputs cannot be reproduced. Sponsors who accelerate with AI but cannot defend the chain of derivation are accumulating a regulatory debt that comes due during inspection, not before.

    Second, audit blindness when programs fail. Most clinical and preclinical programs fail. The institutional value of a failed program is the post-mortem — what did we believe, why did we believe it, where did the evidence break. When AI-derived intermediates cannot be reconstructed, the post-mortem cannot be performed honestly. The organization loses the ability to learn from its own failures, which is the most expensive form of institutional damage.

    Third, decision velocity outrunning evidence integrity. AI compresses cycle times, which is the point. But the same compression means provenance debt accrues faster than humans can repair it. A team can make twelve AI-assisted decisions in the time it used to make one, with one-twelfth the lineage discipline. The danger is not that any one decision is wrong. It is that the organization can no longer tell which decisions rest on which evidence.

    The fix is not more logging. Logging is observational and lossy. Deterministic provenance has to be a property of the system that produces results, not a record kept alongside it — data, code, and computation tracked in the same structure, with versioning and lineage that are queryable and reproducible. Designed in, not bolted on.

    Leaders evaluating AI tooling should ask a single question of every vendor and every internal team: if a regulator, a partner, or a future post-mortem asks how this result was produced, can we rebuild it from if a regulator, a partner, or afuture post-mortem asks how this result was produced, can we rebuild it from first inputs, without depending on the person who ran it? If the honest answer is no, the institution is not adopting AI. It is borrowing against its own credibility.

    The organizations that win the next decade of scientific AI will not be the ones with the largest models. They will be the ones whose AI outputs are still defensible three years after the analyst has left.


    This is the conviction behind how we built the DataJoint platform: provenance as a first-class property of scientific computation, not alogbook kept beside it.

  • DataJoint Enables Seamless Migration of CWL Pipelines to Its Governed, Reproducible Scientific Data Infrastructure

    DataJoint Enables Seamless Migration of CWL Pipelines to Its Governed, Reproducible Scientific Data Infrastructure

    DataJoint today announced native support for converting Common Workflow Language (CWL) pipelines into DataJoint pipelines, enabling research organizations to immediately modernize existing scientific workflows — without sacrificing prior investment or starting from scratch.


    CWL: Widely Adopted, But Increasingly Constrained

    Common Workflow Language has become a de facto standard across pharmaceutical R&D, genomics, and academic research for defining portable, reproducible computational workflows. Major cloud and bioinformatics platforms support CWL natively, and it is broadly adopted across federally funded genomics programs and industry R&D consortia — making it one of the most widely deployed workflow standards in life sciences.

    Yet CWL has recognized limitations in production environments: limited error handling and debugging, no native provenance tracking, poor support for partial re-runs when a step fails mid-pipeline, and no mechanism to query workflow state. As AI-driven research demands tighter auditability and reproducibility, these gaps create real scientific and operational risk.

    What DataJoint Provides

    DataJoint’s CWL conversion layer reads existing CWL workflow definitions and executes them as native DataJoint pipelines. Research teams can extend these pipelines — mixing CWL definitions with DataJoint’s Python-based schema framework — and run them in interpreted mode today, with compiled execution on the roadmap. Key capabilities include:

    Automatic provenance. Every CWL step is backed by DataJoint’s schema-driven provenance layer, creating a complete, queryable record of inputs, outputs, and computational history.

    Granular retry and resilience. Failed steps can be individually retried or corrected without re-running the entire pipeline — a critical capability for long-running, high-cost workflows.

    Queryable state. Workflow state is accessible via DataJoint’s standard query syntax, enabling real-time monitoring and downstream analysis.

    Natural parallelization. Pipelines are decomposed into discrete, independently executable steps that support cluster-level parallelism and graceful pause/resume without lost progress.

    Structured entity database. Critically, DataJoint does not simply execute CWL workflows — it builds a structured database around the scientific entities those workflows produce. The conversion process involves explicitly defining the entities created at each stage (such as processed samples, imaging results, or analysis outputs) and the dependencies between them. This transforms a pipeline from a sequence of compute steps into a living, queryable scientific record — one that captures not just what ran, but what was produced, how it relates to other data, and how it can be reused.

    “Scientific AI will only be as trustworthy as the data foundation beneath it. CWL gave the research community a powerful way to define workflows — DataJoint gives those workflows the provenance, traceability, and governance they need to support defensible science and AI-ready research at scale.” — Jim Olson, CEO, DataJoint

  • DataJoint at CoSyNe 2026: Building AI-Ready Data Workflows for Neuroscience

    DataJoint at CoSyNe 2026: Building AI-Ready Data Workflows for Neuroscience

    This March, DataJoint’s Chief Science Officer Dimitri Yatsenko, PhD, and SciOps Engineer Milagros Marín, PhD, presented a tutorial at CoSyNe 2026 (Computational and Systems Neuroscience) called Building AI-Ready Data Workflows for Neuroscience Experiments.

    The materials from this talk are now available to the public at the conclusion of this blog post.


    The session brought together computational and systems neuroscientists for a hands-on look at what it takes to make scientific data infrastructure ready for AI — not someday, but now.

    Here are the key ideas we shared:

    1. Operational rigor is the foundation for AI in science. Dimitri opened with a provocation: How must research teams transform their work to harness AI? The answer isn’t better models — it’s better data discipline. Without structured schemas, enforced provenance, and reproducible computations, AI agents have nothing reliable to work with. We built on the SciOps Capability Maturity Model — a five-level roadmap from ad hoc scripts to closed-loop AI-assisted discovery — giving labs a concrete path to assess and grow their operational readiness.

    Dr. Dimitri Yatsenko presenting the DataJoint RNA-Seq pipeline in collaboration with the Cadwell Lab at UCSF

    2. The schema is not a record of the science it is the science. We showed how DataJoint’s relational workflow model unifies database, code, and computation into a single formal schema. Tables represent workflow steps, rows represent artifacts, and foreign keys prescribe execution order. The pipeline diagram is the database, not documentation that drifts from reality.

    DataJoint platform architecture. The open-source Python library provides the relational workflow model—schema definition, query algebra, and distributed computation. This core integrates with a relational database (system of record), object storage (for scalable data), and code repositories (for version-controlled pipeline definitions). The managed platform adds infrastructure, observability, and orchestration for production deployments. Milagros’ adaptation from Yatsenko & Nguyen, arXiv:2602.16585 (2026)

    3. Three production pipelines, three scales, one platform. Milagros walked through three real-world projects running on DataJoint: ORION pipelines (brain organoids generation with four electrophysiology modalities integrated, tracking complete provenance from iPSC to spike waveform, in collaboration with the Shcheglovitov lab at the University of Utah); Project AEON (24/7 continuous behavior at the Sainsbury Wellcome Centre, processing 7 million data points per day and weeks of Neuropixels recordings); and DatJoint MoSeq pipeline (unsupervised behavioral syllable discovery in collaboration with the Datta Lab at Harvard Medical School).

    Dr. Milagros Marín demonstrates how DataJoint orchestrates the AEON foraging platform at UCL’s Sainsbury Wellcome Centre — unifying Bonsai-acquired data streams, SLEAP pose estimation, and continuous electrophysiology into a structured, automated pipeline for weeks-long freely moving behavior experiments.

    4. Reproducibility validated, not just claimed. Each project included rigorous validation — positive and negative controls for ORION, dynamic schema generation tested at scale for AEON, and benchmark-matched syllable durations for MoSeq. The pipeline reproduces the science, not just the workflow.

    5. AI agents can query and reason over structured pipelines. We demonstrated an AI assistant that connects directly to a DataJoint pipeline, queries behavioral data, interprets distributions, and generates scientific summaries — all made possible by a self-documenting, queryable schema. A

    As Milagros put it: “Scientists direct, AI agents execute, and the data infrastructure doesn’t just store science — it understands it.”

    6. Open-source, community-ready, publication-grade. All three project codebases are open-source. The ORION pipeline has a paper in preparation (Marín et al., 2026), and a poster will be presented at FENS Forum 2026, and AEON’s preprint (Campagner et al. 2025) is out. We’re building toward an ecosystem where any lab can adopt these workflows and plug in their own protocols.

    DataJoint Tutorial at CoSyNe 2026

    It was energizing to connect with the CoSyNe community — researchers who think deeply about computation and are ready to bring that rigor to their data infrastructure. The conversation reinforced something we believe strongly: the lab of the future doesn’t just manage files — it manages knowledge.


    Missed the tutorial?

    DataJoint Tutorial at datajoint.com/cosyne-2026 — including live demos of all three pipelines and DataJoint’s framework for AI-ready research operations.

    Want to explore what AI-ready workflows look like for your lab? Visit https://docs.datajoint.com or write me an email at milagros@datajoint.com to build AI-ready infrastructure for labs and institutions.

    Trusted Data. Trusted AI. Trusted Science.