TL;DR
Amazon EC2 instances of the same "type" aren't actually identical under the hood; they run on a mix of hardware. This paper reveals that this hardware diversity leads to significant performance variations (up to 60%) within the same instance type, and by strategically selecting better-performing instances, users can achieve substantial cost savings (up to 30%).
What Problem Does This Paper Address?
Cloud computing, particularly services like Amazon EC2, offers a seemingly uniform experience to users. When you request a "large" instance, you expect a certain level of performance. However, as cloud providers upgrade and replace hardware, the underlying infrastructure for a given instance type can become heterogeneous. This paper investigates whether this hardware diversity within the same EC2 instance type leads to noticeable performance variations and, crucially, if users can leverage this heterogeneity to their advantage. The core problem is understanding and quantifying the impact of hidden hardware differences on cloud performance and cost.
Methods
The researchers employed a multi-pronged approach:
- Hardware Identification: They used the
cpuidcommand and/proc/cpuinfoto identify the specific CPU models powering EC2 instances. This allowed them to detect the presence of different hardware configurations within the same instance type. - Micro-benchmarking: To isolate and measure the performance of different hardware components, they used standard micro-benchmarks:
- UnixBench: For CPU performance.
- Redis: For memory performance (specifically GET operations).
- Dbench: For disk subsystem performance.
- Application-level Benchmarking: To see how these micro-level differences translate to real-world applications, they used Httperf to simulate a CPU-bound web server workload.
- Cost Analysis: They developed a cost model to quantify the potential savings achievable by selecting better-performing instances over randomly assigned ones. This model accounts for instance hourly costs, performance differences, and the potential "trial and error" cost of finding optimal instances.
Datasets
The paper does not explicitly mention using pre-existing datasets. Instead, it relies on:
- Measurements from Amazon EC2: The researchers collected hardware information and performance data directly from instances provisioned on Amazon EC2.
- Standard Benchmarking Tools: They utilized established tools like UnixBench, Redis, Dbench, and Httperf, which are commonly used for performance evaluation.
Key Insights and Ideas
- "Same" Instance Type, Different Hardware: The most striking finding is that instances advertised as the same type (e.g.,
m1.large) are indeed hosted on diverse hardware configurations, specifically different CPU models. - Performance Variation is Real and Significant: This hardware diversity translates directly into performance differences. While variations within the same hardware subtype are small, the differences between hardware subtypes within the same instance type can be substantial, reaching up to 60% in some cases.
- Strategic Selection Pays Off: Users aren't just getting a random slice of performance. By understanding the underlying hardware and its performance characteristics, they can actively choose instances that are faster for their specific workload, leading to significant cost savings.
- Hardware Evolution: The study observed that newer CPU models are gradually replacing older ones, and older hardware tends to be relegated to smaller instance types. This highlights the dynamic nature of cloud infrastructure.
Key Results
- CPU Performance: UnixBench showed performance variations of up to 1.21 times between different CPU subtypes within the
m1.largeinstance type. - Memory Performance: Redis benchmarks indicated that faster instances could achieve up to 1.5 times the performance of slower ones.
- Disk Performance: Dbench revealed that some instances offered up to 1.25 times the disk throughput of others.
- Application-Level Performance: The Httperf results showed that the accumulated advantages from faster CPU, memory, and disk subsystems made some instances up to 1.6 times as efficient as others.
- Cost Savings: The cost analysis demonstrated that by selecting better-performing instances, users could achieve up to 30% cost saving. For a specific scenario requiring 100
m1.largeinstances for a year, this translated to a potential saving of $40,664 (16% of the total cost). The paper also identified a threshold of approximately 18 hours of usage beyond which selecting faster instances becomes cost-effective.
Limitations and Future Work
- Limited Instance Types: The study primarily focused on the
m1.largeinstance type, although they noted similar trends inm1.smallandm1.xlarge. Further investigation across a wider range of instance families would be beneficial. - Dynamic Nature of Cloud: The hardware configurations observed were snapshots in time (2011 and 2012). Cloud providers continuously update their infrastructure, so these specific hardware distributions might have changed.
- "Trial and Error" Cost: While the paper models the cost of finding better instances, the exact cost can vary. The assumption of "less than one hour usage is rounded up to and charged as one hour" is a simplification.
- Future Work: The authors suggest investigating the impact of scheduling mechanisms and how they interact with hardware heterogeneity on overall performance. They also propose analyzing the performance of different instance families and availability zones in more detail.
Summary updated