VMThunder: Fast Provisioning of Large-Scale Virtual Machine Clusters

Zhaoning Zhang, Ziyang Li, Kui Wu, Dongsheng Li, Huiba Li, Yuxing Peng, Xicheng Lu

Introduction

In Infrastructure as a Service (IaaS) environments, the ability to rapidly provision virtual machines (VMs) is a critical performance metric. However, when users request hundreds of VMs simultaneously, traditional storage architectures often fail. Standard methods involve copying entire multi-gigabyte VM images from a central server to every compute node, which saturates network bandwidth and overwhelms storage backends. This results in provisioning times that can stretch from minutes to hours.

VMThunder addresses this bottleneck by shifting from a "copying" paradigm to an "on-demand streaming" paradigm. By combining a tree-based peer-to-peer (P2P) architecture with intelligent local caching, VMThunder enables near-instantaneous boot times for large-scale clusters without requiring modifications to the hypervisor or complex image pre-processing.

System Design and Methodology

The core of VMThunder is a decentralized data distribution network that organizes compute nodes into a tree topology. This structure prevents the central storage server from becoming a single point of congestion.

VMThunder System Architecture

Core Mechanisms

  • On-Demand Neighbor Streaming: Instead of downloading an entire image, VMThunder only fetches specific data blocks as they are requested by the VM's boot process.
  • Tree-Based P2P (Relay on Cache): When a node fetches a block from its parent, it caches that block locally. Subsequent requests for the same block from its "children" nodes are served directly from this local cache, offloading the upstream network.
  • Snapshot on Local (SoL): To ensure data consistency and performance, VMThunder uses redirect-on-write snapshots locally. All write operations from the VM are stored on the local host, ensuring that the shared base image remains read-only and the central server is not burdened by VM write traffic.
  • Fault Tolerance: The system employs a multipath scheme. If a parent node fails, children can automatically reroute their requests to "grandparent" nodes or the root storage, ensuring the cluster remains operational.

Experimental Evaluation

The researchers evaluated VMThunder using a diverse set of operating system images, including various Linux distributions (CentOS, Ubuntu, Fedora, OpenSUSE) and Windows Server 2008. Performance was measured using standard I/O benchmarks like Filebench and FIO.

Key Results

  • Rapid Deployment: VMThunder successfully deployed 160 VMs across 160 hosts in just 17 seconds. This is remarkably close to the "ideal" time of 16 seconds (booting from a local disk where the image is already present).
  • Scalability: In comparison to Eucalyptus, a popular open-source cloud platform which took 10 to 20 minutes to provision a similar cluster, VMThunder maintained sub-20-second performance.
  • I/O Integrity: Despite the multi-hop relay chain, runtime I/O performance remained comparable to local LVM disks due to the efficiency of the two-layer local cache.

Key Insights and Discussion

A pivotal discovery driving VMThunder's efficiency is the "10% Observation." Empirical analysis showed that during the boot phase, a VM typically only accesses a small fraction of its total disk image.

Startup Fetch Data vs. Total Disk Usage

As shown in the figure above, while a system disk might occupy several gigabytes, the actual data required to reach a ready state is often less than 10% of that total. Fetching the remaining 90% during the provisioning phase is a significant waste of resources. Furthermore, because VMs running the same OS follow deterministic data access patterns during boot, the P2P caching mechanism becomes highly effective; a parent node is almost certain to have already cached the blocks its children are about to request.

Conclusion

VMThunder demonstrates that large-scale VM provisioning can be achieved in seconds by leveraging on-demand streaming and P2P distribution. By focusing on the small subset of data actually required for system startup and utilizing local resources for write operations, the system eliminates traditional storage bottlenecks. Future work will focus on mitigating "stragglers" (the small percentage of VMs that boot slower than the average) and implementing dynamic topology adjustments to further optimize network performance.

Summary updated

Abstract

Infrastructure as a service (IaaS) allows users to rent resources from the Cloud to meet their various computing requirements. The pay-as-you-use model, however, poses a nontrivial technical challenge to the IaaS cloud service providers: how to fast provision a large number of virtual machines (VMs) to meet users' dynamic computing requests? We address this challenge with VMThunder, a new VM provisioning tool, which downloads data blockson demandduring the VM booting process and speeds up VM image streaming by strategically integrating peer-to-peer (P2P) streaming techniques with enhanced optimization schemes such as transfer on demand, cache on read, snapshot on local, and relay on cache. In particular, VMThunder stores the original images in a share storage and in the meantime it adopts a tree-based P2P streaming scheme so that common image blocks are cached and reused across the nodes in the cluster. We implement VMThunder in CentOS Linux and thoroughly test its performance. Comprehensive experimental results show that VMThunder outperforms the state-of-the-art VM provisioning methods, with respect to scalability, latency, and VM runtime I/O performance.
BibTeX
@article{zhang2014vmthunder,
  author = {Zhaoning Zhang and Ziyang Li and Kui Wu and Dongsheng Li and Huiba Li and Yuxing Peng and Xicheng Lu},
  title = {{VMThunder: Fast Provisioning of Large-Scale Virtual Machine Clusters}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2014},
  doi = {10.1109/tpds.2014.7},
}
Figure