Abstract
In materials science, "long tail" data—small, frequent datasets from daily experiments—often lacks structured provenance, leading to reproducibility challenges. This paper introduces ProvLet, a specialized provenance management service that tracks data lineage and network activity. By utilizing high-level scientific abstractions and an adaptive storage algorithm, ProvLet maintains a robust audit trail with negligible storage overhead (0.002% relative to primary data).
Introduction
Research in fields like materials science frequently generates "long tail" data through instruments such as Scanning Electron Microscopes (SEM). Unlike large-scale "big data" projects, this data is often fragmented, transferred via insecure methods, and documented inconsistently. Existing provenance systems often focus on low-level file changes, which can lead to scalability issues and fail to capture the broader scientific context or network-level security metadata essential for lab forensic audits.
System Architecture and Methods
ProvLet is designed to integrate with the Clowder data management system, built upon three core pillars:
- High-Level Abstractions: Provenance is organized around scientific entities: Spaces (projects), Collections (experimental groups), and Datasets (sample-specific files).
- Dual-Stream Collection: The system captures both data-related logs from Clowder and network-related logs (IP/MAC addresses, protocols) from campus monitoring tools.
- Adaptive Storage Algorithm: To manage log growth, ProvLet employs a priority-based algorithm. Administrators set a Provenance Data Bounding (PDB) value; when reached, the system archives or discards lower-priority records based on event type and age.
Experimental Validation
The system was evaluated using a real-world dataset from the Micro and Nano Technology Lab at UIUC, spanning six years (2015–2020). The study involved 192 researchers managing 297 GB of data across 2,425 datasets and 24,259 files.
Analysis of the collected provenance data revealed that the total log volume for this six-year period was only 5.7 MB, representing a mere 0.002% overhead. The distribution of event types (Figure 4) shows that high-frequency events, such as creating and updating datasets, are prioritized to maintain the context of active experiments.
Key Insights and Results
- Contextual Longevity: Analysis of "Spaces" confirmed that experiments often span years, necessitating long-term provenance maintenance.
- Efficiency: The adaptive algorithm ensures scalability without sacrificing critical lineage information, even in resource-constrained environments.
- Forensic Utility: By logging network metadata, ProvLet enables administrators to trace data uploads to specific machines, enhancing accountability.
- Visualization: Integration with Grafana allows for real-time monitoring of lab activity and event trends, such as the peak in activity observed in 2018.
Conclusion
ProvLet demonstrates that comprehensive provenance tracking for long-tail microscopy data is both feasible and lightweight. By focusing on scientific abstractions and implementing intelligent storage management, it provides a scalable solution for maintaining the integrity and reproducibility of experimental research.
Summary updated