Unified Data Lineage Across Heterogeneous Systems

23 concepts12 papers1 proposal

In modern computing, data rarely stays in one place. A typical workflow might begin with a web-based discovery tool, move to a spreadsheet for cleaning, transition to a SQL database for processing, and conclude in a Jupyter notebook for modeling. While each of these tools may track its own internal history, the "paper trail" of the data—its provenance—is often severed the moment it crosses from one system to another.

As Freire et al. argue, provenance information is frequently trapped within proprietary silos. This fragmentation prevents researchers and developers from answering a fundamental question: where did this specific result come from, and what happened to it along the way?

The Problem of Metadata Silos

The core challenge in provenance management is not a lack of tracking, but a lack of communication. Many high-performance systems have sophisticated internal lineage mechanisms. For example, Interlandi and Condie demonstrate how fine-grained provenance can be integrated into Apache Spark to trace data with minimal overhead. However, such solutions are often confined to a single execution engine. When data moves across heterogeneous processing frameworks, these continuous traces break, leaving gaps in end-to-end observability.

This "tool-switching" reality, described by Kennedy et al., creates a reproducibility nightmare. If a scientist uses a mix of databases, scripts, and manual steps, the resulting record of data origin becomes a fragmented collection of logs that cannot be easily recombined. Freire and Chirigati emphasize that achieving computational reproducibility requires integrating these disparate types of provenance into a single, cohesive lineage.

Bridging Systems with Semantic Standards

To break down these silos, researchers have turned to the Semantic Web. By using standardized representation models, provenance data can remain interpretable even when shared globally. The community-wide "Provenance Challenge" summarized by Moreau et al. highlighted the early diversity of representation formats and the urgent need for a shared language to reason about the history of digital data.

Several frameworks have emerged to provide this common language:

  • Generic Frameworks: Legendre et al. proposed Prov4J, which uses RDF and SPARQL to decouple provenance logic from core applications, allowing developers to capture data history across distributed sources.
  • Specialized Vocabularies: For data engineering, Freitas et al. introduced Cogs, a vocabulary that extends the Open Provenance Model (OPM) to describe Extract-Transform-Load (ETL) workflows in a machine-readable, interoperable way.
  • Version Control Integration: Tools like Git2PROV, developed by De Nies et al., map version control operations to the W3C PROV standard, allowing the evolution of code and data to be consumed by universal analysis tools.

Harmonizing Heterogeneous Traces

Even when systems adhere to standards like W3C PROV, they often extend them in unique, incompatible ways. This creates a "dialect" problem where two systems speak the same language but use different vocabularies.

The SHARP framework, introduced by Gaignard et al., addresses this by using Linked Data and inference rules to harmonize provenance from different scientific workflow systems (like Galaxy and Taverna). By bridging these gaps, the authors enable researchers to perform complex queries that span across different tools and experimental setups.

Similarly, moving away from implicit, centralized tracking is vital for portability. Grauer et al. note that transitioning to standardized lineage formats is essential for making computational transformations transparent across different research platforms.

Why Interoperability Matters

The goal of these efforts is to create a "Web of Data" where trust is built into the infrastructure. As Zhao et al. explain, understanding the origin and history of data is essential for its effective utilization in a decentralized network. When provenance is interoperable, the history of a digital object is no longer a hidden log file; it becomes a first-class citizen that travels with the data, ensuring that every result can be traced, verified, and reproduced regardless of the tools used to create it.

Key Foundational Work

Introduction (2009) — contributed 2 concept elements to this direction.
The First Provenance Challenge (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_provenance_interoperability_96211846,
  title = {provenance interoperability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/provenance-interoperability/rdr_96211846a3574a3b85e359974f6abf4d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_provenance_interoperability_96211846_v1,
  title = {provenance interoperability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/provenance-interoperability/rdr_96211846a3574a3b85e359974f6abf4d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake