Abstract
Provenance—the documented history of a piece of data—is essential for scientific trust and data integrity, yet it is frequently missing or incomplete in legacy datasets. This paper addresses the "paucity of datasets" available for testing provenance reconstruction tools. Rather than building new evaluation frameworks from scratch, the authors propose repurposing existing benchmark corpora from fields such as plagiarism detection, Wikipedia editing, and video copy detection. By mapping these datasets to the W3C PROV standard, the research demonstrates how existing computer science resources can serve as high-quality "gold standards" for provenance reconstruction.
The Challenge of Provenance Reconstruction
In an ideal scientific environment, every dataset and publication would be accompanied by a comprehensive "paper trail" detailing its creation, modifications, and authorship. In practice, however, provenance is often neglected due to the labor-intensive nature of manual tracking. This leaves researchers with "legacy" data that lacks vital context.
To address this, the field of "provenance reconstruction" has emerged, utilizing forensic techniques to hypothesize the history of data based on available metadata and content. However, the development of these tools is hindered by a lack of standardized evaluation datasets. Without a ground truth to compare against, it is impossible to objectively measure the accuracy of a reconstruction tool. This paper seeks to bridge that gap by identifying and adapting existing datasets that already contain the necessary information to validate provenance hypotheses.
Methodology: Mapping to W3C PROV
The authors conducted a comprehensive survey of datasets used in major computer science evaluation initiatives, including TREC (Text REtrieval Conference) and PAN (Uncovering Plagiarism, Authorship, and Social Software Misuse). To determine if a dataset was "provenance-ready," they analyzed its ability to represent three core classes of information defined by the W3C PROV data model:
- Dependency: The relationships between objects (e.g., identifying that a specific paragraph was derived from a source book).
- Sequence of Operations: A documented chain of activities (e.g., a video being cropped, compressed, and then watermarked).
- Authorship: Clear attribution to an agent (e.g., identifying the specific author of an email or a document edit).
The core contribution of this work is the demonstration that these "foreign" datasets can be translated into the standardized PROV language, making them immediately accessible to provenance researchers.
Surveyed Corpora
The paper identifies ten primary corpora across text, image, and video domains that are suitable for repurposing:
- Textual Data: The survey includes METER (journalism reuse), PAN-PC-12 (plagiarism), Wikipedia co-derivative (edit history), PAN-WVC-11 (vandalism), PAN-AI-11 (Enron email authorship), MLaF (mailing list threads), RTE-7 (textual entailment), and arXiv HEP-PH (citation graphs).
- Image Data: SED 2012 (social event detection and clustering).
- Video Data: CCD at TRECVID 2010 (content-based copy detection).
Among these, PAN-PC-12 and TRECVID 2010 are highlighted as the most promising. These datasets are particularly valuable because they track complex sequences of operations—such as text obfuscation or video transformations—providing a rigorous testbed for reconstruction algorithms.
Key Insights: Plagiarism as Provenance
A significant insight of this research is the realization that plagiarism detection and text reuse tasks are essentially specialized cases of provenance reconstruction. Detecting that a paragraph has been plagiarized is, in effect, reconstructing a piece of its provenance.
Furthermore, the authors argue that these repurposed corpora are often superior to existing provenance-specific datasets. Many current provenance datasets provide only the final graph without the raw data. In contrast, the surveyed corpora provide both the raw content (the text or video) and the transformation logs, which are necessary for testing how well a system can move from raw data back to a provenance graph.
Results and Standardized Mapping
The authors successfully mapped complex real-world scenarios into the W3C PROV standard. For instance, they demonstrated that "artificial plagiarism with high obfuscation" can be modeled as a prov:Activity connecting a source prov:Entity to a derivative prov:Entity.
Key findings from the categorization include:
- PAN-PC-12 serves as a powerhouse for text reconstruction, containing 5,000 cases with five distinct transformation types (e.g., shuffling, synonym replacement).
- TRECVID 2010 stands out for multimedia, offering 10 specific video transformations (such as "picture-in-picture" or "cam-coding") to test a system's ability to trace a file back to its source.
Limitations and Future Work
While repurposing existing datasets is a significant step forward, the authors acknowledge several remaining gaps:
- Cross-Media Provenance: Current benchmarks do not track provenance across different media types (e.g., a text summary derived from a video).
- Semantic Depth: Existing datasets often lack rich semantic information regarding the "why" behind an action or the specific roles of agents beyond simple identifiers.
- The Need for Curation: The authors conclude that while repurposing is efficient, the community will eventually require manually curated, provenance-specific benchmarks to capture the full complexity of scientific workflows.
Future efforts will focus on releasing the extracted provenance graphs from these datasets to the research community, providing a standardized foundation for future testing and development.
Summary updated