A Case Study on Parallel HDF5 Dataset Concatenation for High Energy Physics Data Analysis

Sunwoo Lee, Kaiyuan Hou, Kewei Wang, Saba Sehrish, Marc F. Paterno, James Kowalkowski, Quincey Koziol, Robert B. Ross, Ankit Agrawal, Alok N. Choudhary, Wei-keng Liao

TL;DR

High-energy physics experiments generate massive datasets that need to be consolidated for efficient analysis. This paper presents a case study on optimizing the parallel concatenation of HDF5 files, a common format in the field, by tuning various HDF5 features. The findings offer practical guidance for researchers dealing with similar large-scale data aggregation challenges.

What Problem Does This Paper Address?

High-energy physics (HEP) experiments, like the NOvA neutrino experiment, produce enormous amounts of data, often stored as many small files. For effective analysis, these files need to be aggregated into fewer, larger files. This process, known as dataset concatenation, can be a significant bottleneck, consuming substantial time and resources. Furthermore, if not done correctly, the resulting large files might not be efficient for subsequent parallel analysis. This paper tackles the challenge of optimizing this concatenation process, focusing on reducing aggregation time, improving compression efficiency, and ensuring fast access to the consolidated data.

Methods

The authors investigate the impact of various HDF5 features on parallel dataset concatenation performance. Their approach involves:

  • Dataset Concatenation Workflow: A structured workflow is defined, involving distributing input files, collecting metadata, creating a new shared output file, creating datasets with aggregated sizes, and then reading and writing the concatenated data.
  • HDF5 Feature Tuning: They systematically explore and tune several HDF5 features, including:
  • Metadata I/O Modes: Comparing "in-memory I/O" with "on-the-fly I/O" for metadata collection.
  • Dataset Creation: Evaluating "single-process creation" versus "collective creation" for new datasets.
  • Layouts: Examining "contiguous," "chunked," and "compact" data layouts, especially for zero-size datasets.
  • Compression: Investigating different ZLIB compression levels.
  • Caching: Adjusting metadata cache size and metadata block size.
  • Parallel Read Strategies: Comparing "dataset-based partitioning" and "file-based partitioning."
    • I/O Buffer Size: Analyzing the impact of application-level I/O buffer size.
  • Experimental Setup: Experiments were conducted on the Cori supercomputer using both NOvA Near Detector (ND) and Far Detector (FD) data.

Datasets

The paper uses real experimental data from the NOvA (NuMI Off-axis νe Appearance) experiment. This includes:

  • NOvA Near Detector (ND) data: Consisting of 165 input files.
  • NOvA Far Detector (FD) data: Consisting of 6,400 input files.

These datasets are characterized by a large number of small files, with many datasets being zero-sized, and a high degree of compressibility. The paper does not introduce new benchmark datasets but uses these real-world datasets to evaluate the HDF5 features.

Key Insights and Ideas

  • In-Memory I/O is Crucial for Metadata: For large numbers of files and datasets, loading metadata into memory ("in-memory I/O") dramatically speeds up the initial metadata collection phase compared to reading it on-the-fly.
  • Collective Operations for Efficiency: Collective operations, such as collective dataset creation and collective writes, generally offer better performance and scalability than independent operations for the concatenation task.
  • Chunking is Key for Compression, but Size Matters: While chunking is essential for HDF5 compression, the chunk size needs careful tuning. For 1D datasets, 1 MB chunks provided a good balance, while for larger 2D datasets, sizes between 4 MB and 64 MB offered better compression ratios.
  • Larger Metadata Cache Helps: Increasing the metadata cache size (up to 128 MB in their tests) significantly improved performance, especially for write operations, by reducing memory operations and enabling reuse of cached metadata.
  • File-Based Partitioning Outperforms for Reads: For reading data, a "file-based partitioning" strategy (where each process reads its assigned files independently) was more efficient than "dataset-based partitioning" (where all processes collectively read each dataset).
  • Compression Level Trade-offs: Higher compression levels (like ZLIB level 6 or 8) significantly reduce file size but increase compression time. The optimal level depends on the balance between storage savings and processing time.

Key Results

  • Significant Performance Gains: By tuning HDF5 features, the authors achieved substantial improvements. For example, using a larger metadata cache (128 MB) improved write performance by up to 34.5% for ND files.
  • Reduced Output File Size: The optimized concatenation process, especially with effective compression, resulted in output files that were significantly smaller than the sum of the input files (e.g., reducing 1 TB of ND data to 16.9 TB in compressed output).
  • Scalability: The optimized approach demonstrated good strong-scaling performance, with execution times decreasing as the number of processes increased, particularly for larger 2D datasets. For instance, concatenating 6,400 FD files scaled from 2343.13 seconds (100 processes) down to 611.79 seconds (1,600 processes).
  • Metadata Reduction: The overall metadata size in the concatenated output file was also notably smaller than the sum of metadata from all input files.

Limitations and Future Work

  • Memory Footprint of In-Memory I/O: The "in-memory I/O" for metadata, while fast, can consume a large amount of memory, potentially limiting the number of processes that can run on a node. The authors suggest that prefetching metadata blocks into an internal buffer could be a future optimization.
  • HDF5 Limitations: The paper notes that HDF5 currently has limitations on collective I/O for compressed datasets, allowing only one dataset to be accessed collectively at a time. Relaxing this could further improve performance, especially for concatenating many small datasets.
  • Raw Data Chunk Caching: The authors observed no significant performance improvement from increasing the raw data chunk cache size, as their concatenation workflow did not involve repeated access to the same data chunks.
  • Future Work: The authors suggest that HDF5 could improve by optimizing the handling of zero-size datasets to avoid generating unnecessary chunking metadata and by better aggregating chunks from multiple datasets for writing. They also mention that HDF5 is developing fixes for some of these issues.

Summary updated

Abstract

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. The lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.
BibTeX
@inproceedings{lee2022case,
  author = {Sunwoo Lee and Kaiyuan Hou and Kewei Wang and Saba Sehrish and Marc F. Paterno and James Kowalkowski and Quincey Koziol and Robert B. Ross and Ankit Agrawal and Alok N. Choudhary and Wei-keng Liao},
  title = {{A Case Study on Parallel HDF5 Dataset Concatenation for High Energy Physics Data Analysis}},
  booktitle = {Parallel Comput},
  year = {2022},
  url = {https://openreview.net/forum?id=UKU8ROefywz7},
}
Figure