ProvLet: A Provenance Management Service for Long Tail Microscopy Data

Hessam Moeini, Todd Nicholson, Klara Nahrstedt, Gianni PezzarossiarXiv 2021 · arXiv

Abstract

In materials science, "long tail" data—small, frequent datasets from daily experiments—often lacks structured provenance, leading to reproducibility challenges. This paper introduces ProvLet, a specialized provenance management service that tracks data lineage and network activity. By utilizing high-level scientific abstractions and an adaptive storage algorithm, ProvLet maintains a robust audit trail with negligible storage overhead (0.002% relative to primary data).

Introduction

Research in fields like materials science frequently generates "long tail" data through instruments such as Scanning Electron Microscopes (SEM). Unlike large-scale "big data" projects, this data is often fragmented, transferred via insecure methods, and documented inconsistently. Existing provenance systems often focus on low-level file changes, which can lead to scalability issues and fail to capture the broader scientific context or network-level security metadata essential for lab forensic audits.

System Architecture and Methods

ProvLet is designed to integrate with the Clowder data management system, built upon three core pillars:

  1. High-Level Abstractions: Provenance is organized around scientific entities: Spaces (projects), Collections (experimental groups), and Datasets (sample-specific files).
  2. Dual-Stream Collection: The system captures both data-related logs from Clowder and network-related logs (IP/MAC addresses, protocols) from campus monitoring tools.
  3. Adaptive Storage Algorithm: To manage log growth, ProvLet employs a priority-based algorithm. Administrators set a Provenance Data Bounding (PDB) value; when reached, the system archives or discards lower-priority records based on event type and age.
Figure 3: The ProvLet architecture integrates data and network logs into a unified management service with visualization capabilities.

Experimental Validation

The system was evaluated using a real-world dataset from the Micro and Nano Technology Lab at UIUC, spanning six years (2015–2020). The study involved 192 researchers managing 297 GB of data across 2,425 datasets and 24,259 files.

Analysis of the collected provenance data revealed that the total log volume for this six-year period was only 5.7 MB, representing a mere 0.002% overhead. The distribution of event types (Figure 4) shows that high-frequency events, such as creating and updating datasets, are prioritized to maintain the context of active experiments.

Figure 4: Distribution of event types captured by ProvLet, highlighting the frequency of dataset-related interactions.

Key Insights and Results

  • Contextual Longevity: Analysis of "Spaces" confirmed that experiments often span years, necessitating long-term provenance maintenance.
  • Efficiency: The adaptive algorithm ensures scalability without sacrificing critical lineage information, even in resource-constrained environments.
  • Forensic Utility: By logging network metadata, ProvLet enables administrators to trace data uploads to specific machines, enhancing accountability.
  • Visualization: Integration with Grafana allows for real-time monitoring of lab activity and event trends, such as the peak in activity observed in 2018.

Conclusion

ProvLet demonstrates that comprehensive provenance tracking for long-tail microscopy data is both feasible and lightweight. By focusing on scientific abstractions and implementing intelligent storage management, it provides a scalable solution for maintaining the integrity and reproducibility of experimental research.

Summary updated

Abstract

Provenance management must be present to enhance the overall security and reliability of long-tail microscopy (LTM) data management systems. However, there are challenges in provenance for domains with LTM data. The provenance data need to be collected more frequently, which increases system overheads (in terms of computation and storage) and results in scalability issues. Moreover, in most scientific application domains a provenance solution must consider network-related events as well. Therefore, provenance data in LTM data management systems are highly diverse and must be organized and processed carefully. In this paper, we introduce a novel provenance service, called ProvLet, to collect, distribute, analyze, and visualize provenance data in LTM data management systems. This means (1) we address how to filter and store the desired transactions on disk; (2) we consider a data organization model at higher level data abstractions, suitable for step-by-step scientific experiments, such as datasets and collections, and develop provenance algorithms over these data abstractions, rather than solutions considering low-level abstractions such as files and folders. (3) We utilize ProvLet's log files and visualize provenance information for further forensics explorations. The validation of ProvLet with actual long tail microscopy data, collected over a period of six years, shows a provenance service that yields a low system overhead and enables scalability.
BibTeX
@misc{moeini2021provlet,
  author = {Hessam Moeini and Todd Nicholson and Klara Nahrstedt and Gianni Pezzarossi},
  title = {{ProvLet: A Provenance Management Service for Long Tail Microscopy Data}},
  year = {2021},
  eprint = {2109.10897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10897},
}
Figure