Building the I (Interoperability) of FAIR for Performance Reproducibility of Large-Scale Composable Workflows in RECUP

Bogdan Nicolae, Tanzima Islam, Robert Ross, Hubertus J. J. van Dam, Kevin Assogba, Polina Shpilker, М. И. Титов, Matteo Turilli, Tianle Wang, Ozgur O. Kilic, Shantenu Jha, Line Pouchard2023 · DOI

Abstract: Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.

Introduction

In the realm of High-Performance Computing (HPC), "reproducibility" has traditionally focused on numerical consistency—getting the same answer twice. However, as modern scientific workflows become increasingly "converged," mixing traditional numerical simulations with AI and big data analytics, a new challenge has emerged: performance reproducibility. This refers to the ability to ensure a workflow consumes a consistent amount of time and resources across different runs on the same hardware.

Currently, performance variability is a significant hurdle. Researchers have observed that the same scientific code can run up to four times slower from one day to the next on production systems due to system noise, interconnect congestion, or configuration shifts. Diagnosing these issues is nearly impossible because performance data is either not collected, stored in incompatible silos, or so massive that it becomes unsearchable. The RECUP framework addresses this by applying FAIR principles (Findable, Accessible, Interoperable, and Reusable) to performance metadata.

The RECUP Framework

RECUP is designed as a comprehensive framework that treats performance metadata as a managed lifecycle rather than a collection of static files. It integrates several specialized tools to capture a holistic view of workflow execution.

The RECUP architecture integrates metadata capture from various sources, utilizing Mochi for aggregation and DataStates for management and analysis.

As shown in the architecture above, the framework consists of several key layers:

  • Metadata Capture: Tools like RADICAL Cybertools (RCT/EnTK) manage task execution, while Darshan captures fine-grained I/O performance and Chimbuko detects real-time performance anomalies.
  • Aggregation and Management: Mochi acts as the "glue" layer, providing a publish-subscribe model to aggregate data streams. DataStates then manages the evolution of this metadata, providing versioning, indexing, and efficient querying capabilities.
  • Standardization: The final metadata bundle is packaged using RO-Crate, a standardized format that ensures the data is interoperable with other scientific analysis tools.

Solving the Interoperability Challenge

The core innovation of RECUP is the "interoperability bridge." A major hurdle identified during development was the structural mismatch between different tools. Many HPC execution tools utilize a linear Pipeline-Stage-Task model, whereas FAIR standards like RO-Crate expect a dependency-based Data-flow model. RECUP implements formal mapping to reconcile these worldviews during runtime.

Furthermore, the framework moves away from the "omniscient component" model, where a single tool collects all data at the end of a run. Instead, RECUP advocates for "partial profiles," where each tool manages its own metadata slice, which is then aggregated. This prevents data loss if a job is cancelled or if temporary files are purged prematurely.

Key Insights and Results

Testing was conducted on US Department of Energy (DOE) Leadership Computing Facilities, including the Summit, Theta, and Polaris systems, using applications like ExaLearn and NWChem. Several critical insights emerged:

  1. Beyond Containers: While Docker and Singularity provide software portability, they do not address performance variability. Deep metadata about the execution state is required to understand why performance fluctuates.
  2. The Metadata Bottleneck: Standard data paths (e.g., Database -> JSON -> Pandas) create massive memory overhead for large-scale runs. RECUP bypasses this by using binary ingestion and specialized indexing within DataStates, significantly reducing memory utilization and increasing search speed.
  3. Quantifiable Variance: Preliminary results confirmed that production HPC systems can exhibit up to 4x variation in execution time for identical tasks, underscoring the need for automated diagnostic frameworks.

Conclusion and Future Work

RECUP represents a significant step toward making performance metadata a first-class citizen in scientific computing. By creating translators that allow disparate tools to communicate during runtime, the framework enables a unified view of how data movement and task scheduling impact overall timing. Future work will focus on scaling this "bridge" to support a wider array of simultaneous tools without introducing significant system jitter, ultimately providing an end-to-end solution for performance debugging in exascale environments.

Summary updated

Abstract

Scientific computing communities increasingly run their experiments using complex data- and compute-intensive workflows that utilize distributed and heterogeneous architectures targeting numerical simulations and machine learning, often executed on the Department of Energy Leadership Computing Facilities (LCFs). We argue that a principled, systematic approach to implementing FAIR principles at scale, including fine-grained metadata extraction and organization, can help with the numerous challenges to performance reproducibility posed by such workflows. We extract workflow patterns, propose a set of tools to manage the entire life cycle of performance metadata, and aggregate them in an HPC-ready framework for reproducibility (RECUP). We describe the challenges in making these tools interoperable, preliminary work, and lessons learned from this experiment.
BibTeX
@misc{nicolae2023building,
  author = {Bogdan Nicolae and Tanzima Islam and Robert Ross and Hubertus J. J. van Dam and Kevin Assogba and Polina Shpilker and М. И. Титов and Matteo Turilli and Tianle Wang and Ozgur O. Kilic and Shantenu Jha and Line Pouchard},
  title = {{Building the I (Interoperability) of FAIR for Performance Reproducibility of Large-Scale Composable Workflows in RECUP}},
  year = {2023},
  doi = {10.1109/e-science58273.2023.10254808},
}

Loading related content...

Figure