Cloud versus in-house cluster: evaluating Amazon cluster compute instances for running MPI applications

Yan Zhai, Mingliang Liu, Jidong Zhai, Xiaosong Ma, Wenguang ChenSC State of the Practice Reports 2011 · DOI

TL;DR

This paper evaluates Amazon's new Cluster Compute Instances (CCIs) for running tightly coupled MPI applications, comparing them against traditional HPC clusters. While cloud clusters still lag in communication performance, they are rapidly becoming a cost-effective and feasible option for many HPC tasks, especially for development and testing.

What Problem Does This Paper Address?

The rise of cloud computing has presented a tantalizing alternative for HPC users, offering flexibility and pay-as-you-go access. However, previous studies suggested that cloud-based clusters were significantly underperforming traditional HPC systems, making them unsuitable for tightly coupled parallel applications. This paper investigates whether Amazon's newer, HPC-oriented Cluster Compute Instances (CCIs) have closed this performance gap, making cloud a viable alternative for running demanding scientific applications, and explores other critical factors beyond raw performance, such as cost, reliability, and ease of use.

Methods

The authors conducted a comprehensive evaluation by comparing Amazon EC2 CCIs with a local InfiniBand-connected cluster. Their methodology involved:

  • Performance Benchmarking: They used standard benchmarks like the NAS Parallel Benchmarks (NPB) and micro-benchmarks (Intel MPI Benchmark, IOR) to assess computation, communication, and I/O capabilities.
  • Real-World Applications: Three real-world applications (POP, Grapes, mpiBLAST) were used to evaluate performance on more representative workloads.
  • Detailed Analysis: They performed communication profiling to pinpoint performance bottlenecks and analyzed performance variability over time.
  • Parallel I/O Evaluation: A novel aspect of this study was the evaluation of different parallel file system configurations (PVFS and NFS) on the cloud, a level of customization not typically available on traditional clusters.
  • Cost Analysis: A quantitative cost comparison was performed, factoring in performance differences and acquisition/operational costs of a local cluster.

Datasets

The paper utilized the following datasets/benchmarks:

  • NAS Parallel Benchmarks (NPB): Including the BTIO parallel I/O benchmark, run with class C and D, up to 128 processes.
  • Intel MPI Benchmark (IMB): Used for micro-benchmarking communication primitives like Ping-Pong, MPI_Allreduce, and MPI_Alltoall.
  • IOR: A micro-benchmark for testing aggregate read and write bandwidth of parallel file systems.
  • Real-world Applications:
    • POP (Parallel Ocean Program)
    • Grapes (Global/Regional Assimilation and Prediction System)
    • mpiBLAST (parallel implementation of NCBI BLAST)

Key Insights and Ideas

  • Cloud is Catching Up, But Not There Yet: While Amazon CCIs offer improved performance over older EC2 instances, they still lag behind InfiniBand-connected clusters, particularly in communication latency. This means applications heavily reliant on frequent, small message exchanges will still suffer.
  • Latency is the Bottleneck: The study highlights that communication latency, rather than bandwidth, is the primary limiting factor for MPI applications on EC2 CCIs.
  • Customizable I/O is a Game Changer: The ability to configure parallel file systems (like PVFS) on-demand in the cloud offers significant performance advantages for I/O-intensive applications, a flexibility often unavailable on traditional clusters.
  • Load Imbalance is a Cloud-Specific Issue: A surprising finding was load imbalance in certain benchmarks on EC2 CCIs, attributed to TCP/IP protocol processing on CPU cores rather than dedicated hardware offloading, a problem that can be mitigated with careful scheduling.
  • Cost-Effectiveness is Emerging: Despite performance gaps, the cost-effectiveness of cloud HPC is becoming increasingly apparent, especially when considering the total cost of ownership for local clusters and the elasticity of cloud resources.

Key Results

  • Communication Performance: Amazon EC2 CCIs with 10GbE interconnects showed significant improvement over older EC2 instances but were still up to 10x slower than InfiniBand for MPI operations like MPI_Alltoall. Latency for small messages was particularly poor, around 30 times worse than InfiniBand.
  • NPB Performance: For NPB class D benchmarks, EC2 CCIs showed an average performance lag of 23.1% at 64 processes and 32.0% at 128 processes compared to the local cluster. CPU- and memory-intensive applications (like EP and MG) performed comparably, while I/O-intensive ones (like FT and IS) lagged significantly.
  • Parallel I/O: PVFS on EC2 CCIs significantly outperformed NFS for both read and write bandwidth, with performance scaling with the number of I/O servers. For the BT-IO benchmark, PVFS provided up to 37% improvement in execution time over NFS.
  • Cost Analysis: The study suggests that local cluster utilization levels of around 21-25% are sufficient for cloud (EC2 CCI) to be more cost-effective, considering performance and acquisition costs. This implies that for many users, especially those with variable workloads or needing fewer than a full cluster, cloud is already cheaper.
  • Performance Variability: While overall performance was tolerable, communication and virtual storage showed noticeable variability, posing challenges for tuning and optimization.

Limitations and Future Work

  • Communication Bottleneck: The paper acknowledges that the 10GbE interconnect and its associated TCP/IP processing remain a significant limitation for tightly coupled MPI applications. Future work could explore optimizations or hardware advancements to address this.
  • Load Imbalance Mitigation: The authors identified a load imbalance issue due to interrupt handling and proposed a scheduling workaround. Further investigation into the underlying causes and more robust solutions would be beneficial.
  • I/O Performance Anomalies: The paper notes unexplained oscillations in PVFS write performance with 4 servers and suggests further investigation into I/O performance anomalies.
  • Cost Model Simplification: The cost analysis is simplified, not fully accounting for labor costs, hardware upgrades, or the full complexity of local cluster management. A more detailed economic model could provide deeper insights.
  • Future Work: The authors suggest exploring the impact of root-level access in HPC research and development activities on cloud platforms, and further investigating the performance and cost trade-offs for different types of HPC workloads.

Summary updated

BibTeX
@inproceedings{zhai2011cloud,
  author = {Yan Zhai and Mingliang Liu and Jidong Zhai and Xiaosong Ma and Wenguang Chen},
  title = {{Cloud versus in-house cluster: evaluating Amazon cluster compute instances for running MPI applications}},
  booktitle = {SC State of the Practice Reports},
  year = {2011},
  doi = {10.1145/2063348.2063363},
}
Figure