Analyzing File Access Patterns on Large-Scale HPC Systems: Opportunities for File Prefetching

Ahmad Maroof Karimi, Arnab K. Paul, Jong Youl Choi, Lipeng Wan, Feiyi WangMASCOTS 2023 · OpenReview

Abstract

This paper explores the potential opportunities for implementing file prefetching techniques on large-scale high-performance computing (HPC) systems. Specifically, we investigate the file access patterns of various applications across multiple scientific domains using two years' worth of Darshan I/O traces obtained from the Summit supercomputer. We identify recurring trends and patterns which indicate that prefetching can be effectively leveraged to improve data access performance on HPC systems. This study serves as a valuable reference for system architects and developers in the HPC community, providing insights into the opportunities and challenges associated with enabling file prefetching on large-scale HPC systems.

Introduction

In the realm of High-Performance Computing (HPC), processing power frequently outpaces data delivery capabilities. Despite the deployment of massive parallel file systems like Lustre or IBM Spectrum Scale (GPFS), I/O-intensive applications often encounter significant contention, leading to idle CPU cycles while waiting for disk data.

File prefetching—loading data into memory before it is explicitly requested—is a classic optimization technique. However, implementing it effectively in a multi-tenant, large-scale environment is complex. This study seeks to determine if real-world HPC workloads exhibit sufficient repetition to make a generalized, application-aware prefetching tool viable. By analyzing two years of I/O traces from the Summit supercomputer, we demonstrate that HPC applications exhibit highly predictable file access patterns.

Methodology

The research utilizes a data-driven analysis of Darshan I/O traces, a standard characterization tool on HPC systems. The methodology involved three primary steps:

  1. Workload Mapping: Parsing over a million job logs to group individual runs into distinct "applications" (e.g., multiple simulations of the same climate model).
  2. Candidate File Identification: Filtering for "candidate files" that were either read-only or non-overlapping (read entirely before being written) to ensure prefetching does not load stale data.
  3. Temporal and Scale Analysis: Using a moving window (1 to 60 days) to observe pattern evolution and categorizing jobs by scale ("Small" under 45 nodes vs. "Large" over 45 nodes).

The dataset includes 1,237,937 jobs across 4,087 unique applications from 2020–2021 on the Summit supercomputer, representing 348 Petabytes (PB) of read data.

Key Insights and Results

Repetitive Access Patterns

Scientific research is inherently iterative. Researchers often run the same application hundreds of times with minor parameter adjustments, leading to a high "hit rate" for common files such as libraries, configuration files, and static input datasets. Heatmaps of job runs show strong diagonal patterns, proving that consecutive runs of an application are highly likely to access the same files and transfer similar amounts of data.

Heatmaps showing high correlation in file access patterns across consecutive job runs for six representative applications.

Temporal and Scale Dynamics

The probability of two jobs sharing the same files is significantly higher if they are executed close together in time. This suggests that a prefetching model only needs a job's recent history rather than its entire multi-year record. Furthermore, large-scale jobs showed different temporal correlations than smaller ones; for certain applications, large jobs maintained high consistency in file access even over a 60-day window.

Timing and Prefetching Strategy

While file reads occur throughout a job's runtime, there is a significant "burst" of first-read events at the very beginning of execution. However, many other files are accessed mid-run, as shown in the density plots below.

Density of first-read events over normalized job runtime, highlighting the 'start-heavy' access pattern across different applications and scales.

This distribution indicates that an ideal prefetching tool must be "hybrid," loading some files before the job starts and others while it is already running.

Impact Potential

Out of 4,087 unique applications, 2,143 (over 50%) were identified as strong candidates for I/O prefetching. These candidates accounted for 328 PB of the total 348 PB read during the study period, meaning prefetching could impact the vast majority of the system's total I/O load.

Conclusion

The predictability of HPC workloads offers a significant opportunity to reduce I/O bottlenecks through intelligent prefetching. While challenges such as metadata overhead and trace coverage exist, the high correlation between runs suggests that a transparent, lightweight prefetching tool is a viable path forward. Future work will focus on developing such a tool to accelerate scientific discovery at scale.

Summary updated

Abstract

This paper explores the potential opportunities for implementing file prefetching techniques on large-scale high-performance computing (HPC) systems. Specifically, we investigate the file access patterns of various applications across multiple scientific domains using two years' worth of Darshan I/O traces obtained from the Summit supercomputer. We identify recurring trends and patterns which indicate that prefetching can be effectively leveraged to improve data access performance on HPC systems. This study serves as a valuable reference for system architects and developers in the HPC community, providing insights into the opportunities and challenges associated with enabling file prefetching on large-scale HPC systems.
BibTeX
@inproceedings{karimi2023analyzing,
  author = {Ahmad Maroof Karimi and Arnab K. Paul and Jong Youl Choi and Lipeng Wan and Feiyi Wang},
  title = {{Analyzing File Access Patterns on Large-Scale HPC Systems: Opportunities for File Prefetching}},
  booktitle = {MASCOTS},
  year = {2023},
  url = {https://openreview.net/forum?id=4MHdiUE5qW},
}
Figure