Uncovering Access, Reuse, and Sharing Characteristics of I/O-Intensive Files on Large-Scale Production HPC Systems

Tirthak Patel, Suren Byna, Glenn K. Lockwood, Nicholas J. Wright, Philip H. Carns, Robert B. Ross, Devesh Tiwari

TL;DR

This paper dives deep into the I/O patterns of large-scale High-Performance Computing (HPC) systems, revealing that files are often accessed and reused across different applications and runs, challenging previous assumptions. The study highlights significant variability in I/O time (not data size) across runs, suggesting opportunities for optimizing storage systems and workflow scheduling.

What Problem Does This Paper Address?

Understanding how files are accessed, reused, and shared is crucial for optimizing storage systems in HPC environments. However, collecting this data from production systems is challenging due to the high overhead of traditional tracing methods. This lack of insight hinders our ability to design efficient storage solutions, predict performance bottlenecks, and optimize application workflows. Previous studies have often focused on application-level traces or limited benchmarks, leaving a gap in our understanding of real-world, large-scale file I/O behavior.

Methods

The researchers utilized Darshan, a lightweight I/O monitoring tool, to collect file I/O access patterns on the Cori supercomputer over a four-month period. This approach allowed them to gather data from millions of file accesses across numerous applications and users without significantly impacting system performance. They then developed a methodology to:

  • Identify "I/O-intensive" files: Files that transferred at least 100 GB of data and were accessed more than once.
  • Classify files: Categorizing them as read-heavy (RH), write-heavy (WH), or read-and-write-heavy (RW) based on data transfer patterns.
  • Classify runs: Distinguishing between "read runs" and "write runs" based on whether they predominantly read or wrote data.
  • Analyze inter-arrival times: Measuring the time between consecutive accesses to a file by runs or applications.
  • Examine sharing patterns: Determining how many different applications accessed the same file.
  • Investigate variability: Quantifying the differences in I/O data transfer and I/O time across runs and within runs.

Datasets

The study used a dataset of approximately 84 million Darshan logs collected from the Cori supercomputer over a four-month period (October 2017 - January 2018). This data spanned approximately 52 million unique files, 8489 applications, and 651 users, with a total data transfer of 12.8 PB. No new benchmarks were introduced; the study analyzed real-world production workloads.

Key Insights and Ideas

  • Files are Shared and Reused: Contrary to some assumptions, a significant portion of files (over 67%) are accessed by multiple applications, and these files are often both read from and written to (RW files). This highlights a producer-consumer relationship where files are written by some runs and read by others, potentially across different applications.
  • Runs are Monolithic in I/O Type: A surprising finding is that most application runs are either predominantly read-intensive or write-intensive, rather than performing both phases within a single run. This suggests a shift towards more modular scientific workflows.
  • I/O Time Variability is Key: While the amount of data transferred per file across runs shows little variability, the time it takes to transfer that data exhibits significant variability. This is particularly pronounced for read-heavy files.
  • Temporal Load Imbalance Matters: Peak I/O activity periods (e.g., 3 am-5 am) show higher I/O times but lower variability, while off-peak periods have lower I/O times but higher variability. This suggests that system contention during peak times can mask individual I/O time differences.
  • Spatial Load Imbalance on OSTs: Despite capacity balancing, Object Storage Targets (OSTs) show a large inequality in data transfer, indicating that dynamic load balancing is a critical need for Lustre file systems.

Key Results

  • File Classification: 22% of files are read-heavy (RH), 7% are write-heavy (WH), and 71% are read-and-write-heavy (RW).
  • Run Classification: 69% of runs are read runs, and 31% are write runs.
  • Inter-arrival Times: For 80% of files, re-access occurs after more than 50-55 hours, significantly longer than typical job runtimes.
  • Consecutive Runs: Files experience more consecutive read runs (average >14) than consecutive write runs (average <4).
  • Multi-Application Sharing: Over 67% of files are accessed by at least two applications.
  • Data Transfer per Run: Write runs transfer, on average, 1.4x more data per run than read runs.
  • I/O Time Variability: The mean Coefficient of Variation (CoV) for I/O time across runs is 39%, compared to a mean CoV of 12% for I/O data size. RH files show the highest I/O time variability (mean CoV of 68%).
  • OST Load Imbalance: The least active OST is only 13% as active as the most active OST, despite balanced file and user distribution.

Limitations and Future Work

Limitations:

  • User Opt-Out: Some users could opt out of Darshan logging, potentially missing some I/O activity.
  • Unavailable Information: The study could not analyze file size, file amendments/overwrites, number of nodes involved in I/O, or batch job I/O behavior. Random vs. sequential I/O was largely sequential and thus not insightful.
  • "What if?" Analysis: Retroactively posing hypothetical scenarios (e.g., removing a run) is not possible with the collected data.
  • Environment Specificity: Findings are influenced by the NERSC environment and Cori's specific workloads, cautioning against direct generalization to all HPC systems.

Future Work:

  • The findings suggest opportunities for data compression for files with long inter-arrival times and transparent burst-buffer prefetching/caching for files with shorter expected inactivity.
  • The classification of runs and files can inform file placement decisions in multi-tier storage systems.
  • The insights into consecutive read/write runs can help leverage MPI "hints" for system optimization and partitioning of I/O servers.
  • The significant intra-run variability due to OST load imbalance highlights the need for dynamic file migration, replication of read-only data, and caching in Lustre file systems.
  • The study provides a methodological framework for conducting similar analyses at other HPC centers to confirm or refute these findings.

Summary updated

BibTeX
@inproceedings{patel2020uncovering,
  author = {Tirthak Patel and Suren Byna and Glenn K. Lockwood and Nicholas J. Wright and Philip H. Carns and Robert B. Ross and Devesh Tiwari},
  title = {{Uncovering Access, Reuse, and Sharing Characteristics of I/O-Intensive Files on Large-Scale Production HPC Systems}},
  booktitle = {FAST},
  year = {2020},
  url = {https://openreview.net/forum?id=1Rf2agAXmd},
}
Figure