Automated Provenance Tracking for Verifiable Data Pipelines

This direction is split into other directions: automated provenance tracking. Its last version stays readable and citable here.

Version 1 of 1338 papers482 conceptsfrozen 2026-10-04

In modern science and engineering, we rarely move directly from raw data to a final insight. Instead, we navigate a "perfect storm" of complexity: multi-stage pipelines where data is cleaned, transformed, compressed, and fed into models that are themselves pruned or quantized. When a system fails or a result seems suspicious, we face the "black box" problem. Without a record of how a result was derived, we cannot debug it, reproduce it, or trust it.

The research direction emerging across these papers is the shift toward Automated Provenance Tracking. Rather than treating documentation as a manual, post-hoc chore, these researchers propose embedding a "digital paper trail" directly into the plumbing of our computational systems.

The Pedigree of a Result

The core intuition is that every data object—whether a genomic sequence, a machine learning model, or a safety inspection record—should carry its "pedigree." Ronald R. Sederoff and colleagues demonstrated this with the Expresso system, which tracks the lineage of experimental parameters from lab instruments to data mining.

By recording the "experimental pedigree," researchers ensure that a result isn't just a static value, but a verifiable conclusion linked to specific chemical concentrations and instrument settings. This transforms a result from a "tombstone" (a term used by Frank van Harmelen and Carole Goble in Theoretical and technological building blocks for an innovation accelerator) into a living, citable object.

Infrastructure as the Source of Truth

For provenance to work at scale, it must be automated. Ian T. Foster argues in Human-Machine Symbiosis, 50 Years On that scientists are bogged down by "mechanical" clerical work. The solution is Service-Oriented Science, where the system automatically captures software versions and parameters as a side effect of the workflow.

This "background" capture is now a cornerstone of ML Engineering. In Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX), Jarek Wilkiewicz and his team describe metadata as a "functional equivalent to a software stack trace." If a model in production begins to drift, developers don't guess what went wrong; they walk the lineage back to the specific dataset or transformation that caused the failure.

Granular Lineage and the System Stack Trace

The power of this approach lies in its granularity. It is not enough to know which dataset was used; we need to know what happened to individual data points.

  1. Token-Level Tracking: Systems like ADAMS, developed by Joaquin Vanschoren, embed provenance directly into "data tokens" as they pass through a pipeline.
  2. Transformation Mapping: In model compression, Angie Boggust uses hierarchical visualizations in Compress and Compare to track how pruning and quantization alter model behavior over time.
  3. Document-Level Metadata: For large-scale NLP, Alexander Leonhardt and Giuseppe Abrami use the DUUI framework to record the specific model version used for every single document in a processing run.

By treating explainability as a systems-level tracking problem—an idea championed by Ion Stoica and David A. Patterson in A Berkeley View of Systems Challenges for AI—we turn opaque AI agents into auditable sequences of events.

Verification and Regulatory Trust

Beyond debugging, automated provenance is becoming a requirement for safety and law. Susan Landau 0001 and Steven M. Bellovin point out that for a citizen to challenge a government AI decision, the system must be able to exactly reproduce the state it was in at the moment the decision was made.

To ensure this trail is tamper-proof, researchers are turning to immutable ledgers. Hao Dai and colleagues developed TrialChain, which logs cryptographic hashes of intermediate analysis steps onto a blockchain. Similarly, the CERTAIN project led by Fabian Kovac uses "Semantic MLOps" to provide a verifiable path for regulatory certification under frameworks like the EU AI Act.

Ultimately, this research direction suggests that trust is a property of the process, not just the output. By automating the capture of lineage, we move from a world of "black box" assertions to a world of verifiable, reproducible, and accountable intelligence.

Papers in this version

Showing 60 of 338 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_provenance_tracking_for_verifi_19f2f27b,
  title = {Automated Provenance Tracking for Verifiable Data Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-provenance-tracking-for-verifiable-data-pipelines/rdr_19f2f27be1b443ec91417b5317390d76},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_provenance_tracking_for_verifi_19f2f27b_v1,
  title = {Automated Provenance Tracking for Verifiable Data Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-provenance-tracking-for-verifiable-data-pipelines/rdr_19f2f27be1b443ec91417b5317390d76/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}