VL2: a scalable and flexible data center network

Albert G. Greenberg, James R. Hamilton, Navendu Jain, Srikanth Kandula, Changhoon Kim, Parantap Lahiri, David A. Maltz, Parveen Patel, Sudipta SenguptaCommunications of the ACM 2011 · DOI

TL;DR

This paper introduces VL2, a data center network architecture that provides uniform high capacity, performance isolation, and Layer-2 semantics using commodity hardware and a novel approach to address resolution. VL2 enables unprecedented agility in data center resource allocation, allowing any server to be assigned to any service without compromising performance.

What Problem Does This Paper Address?

Modern cloud services are driving the creation of massive data centers with tens to hundreds of thousands of servers. The key challenge is to make these data centers agile, meaning servers can be dynamically allocated to different services as demand fluctuates or equipment fails. Existing data center network architectures, typically hierarchical and built with expensive hardware, suffer from several limitations:

  • Limited Server-to-Server Capacity: Hierarchical designs lead to oversubscribed links, creating "hot spots" and fragmenting the server pool. This prevents efficient resource utilization, as idle servers in one part of the network cannot easily serve demand in another.
  • Lack of Performance Isolation: A traffic flood in one service can negatively impact other services sharing the same network infrastructure, leading to unpredictable performance.
  • Configuration Burden and Inflexibility: Traditional networks often tie server IP addresses to their physical location, making it difficult to migrate virtual machines or reassign servers to different services without complex reconfigurations. This limits agility and increases operational overhead.

Methods

VL2 tackles these problems by introducing a new network architecture with three core objectives: uniform high capacity, performance isolation, and Layer-2 semantics.

  • Network Topology: VL2 employs a Clos network topology built from low-cost commodity switches. This provides a rich, interconnected backbone with high path diversity between servers.
  • Valiant Load Balancing (VLB): To handle the volatile and unpredictable traffic patterns observed in data centers, VL2 uses VLB. Instead of complex traffic engineering, VLB randomly spreads traffic across available paths. This destination-independent approach helps prevent hot spots and ensures uniform capacity.
  • Layer-2.5 Shim and End-System Based Resolution: VL2 introduces a "shim" layer in the server's network stack. This shim intercepts ARP requests and translates them into queries to a VL2 Directory System. This system maps application-specific IP addresses (AAs) to location-specific IP addresses (LAs) of the ToR switches. This separation of names from locators is crucial for agility, allowing servers to retain their AAs even when migrated.
  • IP-in-IP Encapsulation and ECMP: To implement VLB, VL2 uses IP-in-IP encapsulation. Servers encapsulate their packets with the LA of a randomly chosen intermediate switch. The network then uses Equal-Cost MultiPath (ECMP) forwarding to distribute traffic across multiple paths to that intermediate switch.
  • Directory System: A two-tiered directory system (read-optimized directory servers and write-optimized RSM servers) handles AA-to-LA mappings, providing scalability, reliability, and low latency for lookups.

Datasets

The paper's design and evaluation are based on:

  • Production Data Center Traffic Data: Detailed measurements of traffic and fault data from a large operational cloud service provider were used to understand traffic patterns, flow sizes, and failure characteristics.
  • VL2 Prototype Testbed: A prototype was built and evaluated on an 80-server cluster with 10 commodity switches. This testbed was used to perform experiments simulating various workloads, including all-to-all data shuffles and performance isolation scenarios.

Key Insights and Ideas

  • Agility through Simplicity: The core insight is that achieving high agility in data centers doesn't necessarily require complex, custom hardware or intricate traffic engineering. Instead, a well-designed architecture leveraging commodity components and intelligent end-system behavior can provide the desired flexibility.
  • VLB as a Robust Solution for Volatility: The paper demonstrates that VLB, despite its apparent simplicity, is highly effective in data center environments with unpredictable traffic. Its ability to spread traffic randomly across many paths prevents congestion and ensures uniform capacity, outperforming more complex schemes in practice.
  • Separating Names from Locators is Key: The separation of application addresses (AAs) from network locations (LAs) at the end-system level is fundamental to enabling VM migration and dynamic server allocation without network reconfiguration.
  • Commodity Hardware is Sufficient: VL2 proves that existing, low-cost commodity switches and servers can be used to build a high-performance, scalable data center network, challenging the notion that specialized, expensive hardware is required.
  • End-System Intelligence: Shifting intelligence to the end-system (VL2 agent) allows for greater flexibility and avoids burdening the network control plane with host-specific information, which is crucial for scalability.

Key Results

  • Uniform High Capacity: The VL2 prototype achieved 94% of the maximum possible network capacity during an all-to-all data shuffle of 2.7 TB among 75 servers, completing the task in 395 seconds.
  • High TCP Fairness: A TCP fairness index of 0.995 was achieved, indicating excellent load balancing and minimal congestion.
  • Performance Isolation: Experiments showed that the performance of one service was unaffected by traffic ramps in another service, even when servers were intermingled.
  • Fast Convergence: The network reconverged in less than 1 second after link failures, demonstrating graceful degradation.
  • Scalable Directory System: The directory system could handle 50K lookups/sec with latency under 10ms and 12K updates/sec within 600ms.
  • Cost-Effectiveness: A VL2 network with no over-subscription was found to be comparable in cost to current networks with significant over-subscription, and significantly cheaper than building a non-oversubscribed conventional network.

Limitations and Future Work

  • TCP Reliance for Hose Model: VL2 relies on TCP's end-to-end congestion control to enforce the "hose model" (limiting traffic to available bandwidth). While effective for TCP traffic, the paper acknowledges that UDP traffic might require different handling.
  • VLB vs. Optimal Routing: While VLB is simple and effective, it incurs a slight increase in link utilization (around 20% higher for the most loaded links) compared to optimal adaptive routing schemes. The paper argues this is a worthwhile trade-off for simplicity and robustness.
  • Future Work: The paper suggests further exploration of traffic engineering mechanisms and cost-benefit analyses for different levels of over-subscription. The authors also highlight the potential for VL2 to support more complex operations like data shuffles efficiently, impacting algorithmic and data storage strategies.

Summary updated

BibTeX
@article{greenberg2011vl,
  author = {Albert G. Greenberg and James R. Hamilton and Navendu Jain and Srikanth Kandula and Changhoon Kim and Parantap Lahiri and David A. Maltz and Parveen Patel and Sudipta Sengupta},
  title = {{VL2: a scalable and flexible data center network}},
  journal = {Communications of the ACM},
  year = {2011},
  doi = {10.1145/1897852.1897877},
}
Figure