User-based I/O Profiling for Leadership Scale HPC Workloads

Ahmad Hossein Yazdani, Arnab K. Paul, Ahmad Maroof Karimi, Feiyi Wang, Ali Raza ButtICDCN 2025 · OpenReview

Abstract

Traditional High-Performance Computing (HPC) I/O characterization often focuses on application code or system architecture, neglecting the significant impact of user-induced variance. This research demonstrates that I/O behavior is heavily driven by the "human in the loop." By analyzing a year of Darshan logs from the Summit supercomputer, we show that application I/O patterns are highly predictable (80-100% certainty) when accounting for user-specific command-line arguments and historical job patterns.

Introduction

In leadership-scale supercomputing, I/O contention remains a critical bottleneck. While software and hardware configurations are often scrutinized, the role of the individual user is frequently overlooked. A single application can exhibit vastly different I/O signatures depending on user-defined parameters, such as node allocation or output formats. This study addresses "user-induced variance" by profiling I/O at the job level, identifying distinct signatures tied to specific user-application pairs.

Methodology

The researchers developed a multi-stage framework to characterize and predict I/O behavior using a combination of lightweight instrumentation and unsupervised learning.

Feature Engineering and Data Extraction

The framework utilizes Darshan, a lightweight I/O characterization tool, to capture detailed statistics from various layers of the software stack, including MPI-IO, POSIX, and STDIO.

The Darshan I/O characterization pipeline extracts and persists library-specific information to create job-level profiles.

Over 200 features were extracted per job, categorized by interface, file system, access type, and transfer size. This high-dimensional data allows for a granular view of how data moves through the system.

Summary of key I/O features used to generate user-based profiles, including interface types and file access patterns.

Unsupervised Clustering and Modeling

To identify unique "I/O signatures," the authors applied HDBSCAN (a density-based clustering algorithm) to group jobs by user + application. This approach moves beyond generic application profiling to capture how specific individuals utilize tools. Predictability was then quantified using Shannon Entropy, measuring the uncertainty of I/O patterns relative to command-line arguments. Finally, Mutual Information was employed to assess the temporal correlation between a user's past and future job configurations.

Dataset and Scope

The study analyzed a massive dataset from the Summit supercomputer at Oak Ridge National Laboratory, covering the entire year of 2020. The analysis included approximately 93,000 job submissions from 38 high-activity users across 50 diverse applications, including well-known HPC benchmarks like LAMMPS and E3SM.

Key Results and Insights

The analysis revealed several critical findings regarding the nature of HPC workloads:

  • The Human Signature: I/O behavior is not a static property of software. Different users running the same binary often produce entirely distinct I/O clusters, reflecting different research goals or configurations.
  • High Predictability: Command-line arguments serve as excellent proxies for I/O behavior. For major applications, I/O pattern uncertainty often dropped to near zero when historical command-line data from the previous 24 hours was available.
  • Temporal Locality: Users typically work in "sprints," showing high correlation between current and past job configurations within a 1-to-10-day window. However, this predictability window varies significantly between individual users.
  • Hidden Configuration Knobs: Internal simulation variables, often not explicitly visible in simple logs, can drastically shift I/O patterns from small, frequent operations to massive, infrequent transfers.

Conclusion and Future Directions

This work highlights the necessity of incorporating user context into I/O scheduling and system management. By recognizing that users are often the primary drivers of I/O variance, system administrators can better anticipate load. Future research may leverage Large Language Models (LLMs) to parse complex submission scripts and source code for deeper configuration insights. Ultimately, these predictive models could be integrated into real-time I/O schedulers to anticipate heavy traffic and mitigate system-wide contention before it impacts performance.

Summary updated

BibTeX
@inproceedings{yazdani2025user,
  author = {Ahmad Hossein Yazdani and Arnab K. Paul and Ahmad Maroof Karimi and Feiyi Wang and Ali Raza Butt},
  title = {{User-based I/O Profiling for Leadership Scale HPC Workloads}},
  booktitle = {ICDCN},
  year = {2025},
  url = {https://openreview.net/forum?id=63jxb9gpzZ},
}
Figure