One optimized I/O configuration per HPC application: leveraging the configurability of cloud

Mingliang Liu, Jidong Zhai, Yan Zhai, Xiaosong Ma, Wenguang ChenAPSys 2011 · DOI

TL;DR

This paper explores how the flexibility of cloud platforms can be leveraged to optimize I/O performance for High Performance Computing (HPC) applications. By tailoring I/O configurations (like file systems and storage devices) to specific application needs, significant performance and cost improvements can be achieved, demonstrating that a "one-size-fits-all" approach is suboptimal in the cloud.

What Problem Does This Paper Address?

High Performance Computing (HPC) applications are not only computationally intensive but also heavily reliant on efficient I/O and network performance. Migrating these applications to cloud platforms, such as Amazon EC2 Cluster Compute Instances (CCIs), presents a challenge. While previous research has focused on the impact of virtualization and interconnects, this paper highlights a crucial, yet often overlooked, aspect: the configurability of cloud I/O subsystems. Traditional HPC systems offer fixed configurations, but cloud environments allow users to dynamically adjust parameters like file systems, the number of I/O nodes, and virtual disk types. The paper aims to demonstrate that this configurability can be exploited to create customized, balanced I/O subsystems that significantly boost the performance and cost-effectiveness of I/O-intensive HPC applications.

Methods

The authors investigate the impact of I/O configuration on HPC applications by leveraging the flexibility of Amazon EC2 CCIs. Their approach involves:

  • File System Selection: Evaluating different parallel and shared file systems, specifically focusing on NFS (Network File System) and PVFS (Parallel Virtual File System), to understand their performance characteristics for HPC workloads.
  • Storage Device Selection: Comparing the performance of ephemeral disks (local, non-persistent) and Elastic Block Store (EBS) volumes (network-attached, persistent) for I/O operations.
  • File System Internal Parameter Tuning: Examining the effect of internal file system parameters, such as synchronous vs. asynchronous write modes for NFS, and the impact of combining multiple EBS disks into a software RAID configuration.
  • I/O Server Placement: Investigating two I/O server placement strategies: dedicated instances (separate compute nodes) and part-time instances (co-located with compute processes).
  • Benchmarking: Using the IOR (Interleaved-Object-Reader) micro-benchmark to evaluate raw storage performance and two representative HPC applications: BTIO (an I/O-intensive benchmark from the NAS Parallel Benchmarks suite) and POP (an ocean circulation model).

Datasets

The paper does not explicitly mention specific datasets used for the HPC applications. Instead, it focuses on the performance of the applications themselves with simulated or generated I/O patterns.

  • IOR Benchmark: Used to measure raw I/O bandwidth with varying block sizes and configurations.
  • BTIO Benchmark: A class C problem size was used, generating an output file of approximately 6.4GB.
  • POP Application: Used a grid dimension of 320x384x20, producing an aggregated output of about 6GB.

Key Insights and Ideas

  • Cloud Configurability is Key: The primary takeaway is that the ability to customize I/O subsystems in the cloud is a significant advantage over traditional HPC systems. This allows for per-application optimization, rather than a one-size-fits-all approach.
  • Application-Specific I/O Matters: Different HPC applications have distinct I/O requirements. What works well for one application might be suboptimal for another, leading to substantial performance or cost differences (up to 2.5-fold observed).
  • PVFS vs. NFS Trade-offs: For I/O-intensive applications like BTIO, PVFS generally outperformed NFS, especially when scaled with multiple I/O servers. However, for applications with a single process handling all I/O (like POP), NFS with async writes could be more effective.
  • Ephemeral vs. EBS: While the performance difference between ephemeral and EBS disks was found to be small for writes, ephemeral disks offered better availability and lower performance variance. The choice depends on whether persistent storage is required.
  • I/O Server Placement Impacts Cost and Performance: Using part-time I/O servers (co-located with compute nodes) can offer cost benefits, though performance might be affected by resource contention.

Key Results

  • BTIO Performance: PVFS significantly outperformed NFS for BTIO, showing up to a 60% performance improvement when using PVFS with multiple dedicated servers compared to NFS with one dedicated server.
  • POP Performance: For POP, NFS (with async mode) outperformed PVFS across all tested configurations.
  • Cost Savings: Figure 4 illustrates that using part-time I/O servers can lead to cost savings, and at small scales, NFS with a part-time server was the most cost-effective option.
  • NFS Write Modes: Asynchronous write mode for NFS offered a significant performance advantage over synchronous mode, especially with small data blocks.
  • Disk Scaling Limitations: For NFS with EBS disks, the system hit peak bandwidth with only two disks, suggesting that simply adding more disks doesn't always yield proportional performance gains due to underlying cloud infrastructure complexities.

Limitations and Future Work

  • Limited Scope of Applications: The study evaluated only two specific HPC applications. The authors acknowledge that a broader range of applications with diverse I/O patterns would be needed for more comprehensive conclusions.
  • EBS Availability Issues: The experiments were conducted using ephemeral disks due to "unstable availability of EBS storage" encountered during testing. This limits direct comparison of EBS performance in all scenarios.
  • Disk Scaling Investigation: The paper notes that the disk scaling problem (optimizing the number of disks per I/O server) requires further investigation.
  • Automated Configuration: The authors express a future goal to explore how storage options interact with HPC application characteristics and to automate the process of choosing optimal I/O storage configurations for individual applications.
  • Interconnect Limitations: The paper briefly mentions that BTIO's computation component did not scale well on Amazon EC2 due to unsatisfactory interconnection, which indirectly affects the perceived impact of I/O optimizations.

Summary updated

BibTeX
@inproceedings{liu2011one,
  author = {Mingliang Liu and Jidong Zhai and Yan Zhai and Xiaosong Ma and Wenguang Chen},
  title = {{One optimized I/O configuration per HPC application: leveraging the configurability of cloud}},
  booktitle = {APSys},
  year = {2011},
  doi = {10.1145/2103799.2103818},
}

Loading related content...

Figure