PolarFly: A Cost-Effective and Flexible Low-Diameter Topology

Kartik Lakhotia, Maciej Besta, Laura Monroe, Kelly Isham, Patrick Iff, Torsten Hoefler, Fabrizio Petrini

Introduction

In the era of large-scale AI models and complex scientific simulations, the network interconnect has emerged as a primary bottleneck for system performance. As the number of GPUs and CPUs increases, the latency and throughput of the communication fabric determine the overall efficiency of the cluster. To minimize latency, researchers have long sought "low-diameter" topologies—networks where the maximum number of hops between any two nodes is kept to a minimum.

PolarFly represents a significant advancement in this field. It is a diameter-2 network topology that pushes the boundaries of node density and cost-effectiveness. By leveraging the mathematics of Erdős-Rényi (ER) polarity graphs, PolarFly achieves over 96% of the theoretical Moore bound efficiency. Unlike its predecessors, PolarFly is designed with practical data center constraints in mind, offering a modular and expandable architecture that aligns with commercial hardware.

The Challenge of Diameter-2 Networks

The "Gold Standard" for low-latency networking is a diameter-2 topology, where any two nodes are separated by at most two hops. While mathematically desirable, designing such a network that is both high-density (connecting the maximum number of nodes for a given number of router ports) and flexible is a significant challenge.

The previous state-of-the-art, Slim Fly, was mathematically optimized for density but suffered from "rigidity." Slim Fly supported only a limited set of router port counts (radixes) and lacked a clear path for incremental expansion. Adding nodes to a Slim Fly network often required a complete rewiring of the data center. PolarFly addresses these limitations by providing a design that is both dense and adaptable to real-world deployment scenarios.

Methodology and Mathematical Foundation

PolarFly is derived from the Erdős-Rényi family of polarity graphs found in finite geometry. The topology utilizes dot products over finite fields ($\mathbb{F}_q$) to determine connectivity.

Orthogonality and Connectivity

In the PolarFly model, nodes are represented as vectors. An edge exists between two nodes if their vectors are "orthogonal" (their dot product is zero). This mathematical property ensures that every pair of non-connected nodes shares exactly one common neighbor, which is the fundamental requirement for maintaining a diameter of 2 while maximizing node density.

Modular Layout

The authors introduce a modular layout by grouping nodes into "clusters" or racks. The design identifies two distinct types of clusters:

  1. Quadric Clusters: Containing nodes that are orthogonal to themselves.
  2. Non-Quadric Clusters: Containing nodes that are not self-orthogonal.

This distinction allows for a structured cabling approach that is more manageable than purely random or unstructured low-diameter graphs.

Incremental Expansion

One of PolarFly's most significant contributions is its support for incremental growth. The paper proposes two "cluster replication" methods that allow operators to add new racks of nodes without touching existing cables. These methods either maintain the strict diameter-2 property for moderate growth or transition to a diameter-3 property for massive scaling, offering a flexibility previously unavailable in high-density topologies.

Adaptive Routing ($UGAL_{PF}$)

To handle congestion, the authors developed $UGAL_{PF}$, a specialized version of Universal Globally-Adaptive Load-balancing. This routing scheme utilizes "Compact Valiant" routing, which allows the network to bypass congested links using 3-hop paths. This is more efficient than the 4-hop paths typically required by standard Valiant routing in other topologies.

Evaluation and Results

The performance of PolarFly was evaluated using BookSim, a cycle-accurate simulator, against several baselines including Slim Fly, Dragonfly, Jellyfish, and 3-level Fat Trees. The evaluation utilized synthetic benchmarks mimicking real-world workloads, such as Uniform Random, Tornado, and Random Permutation traffic.

Efficiency and Hardware Mapping

PolarFly reaches >96% of the Moore bound, significantly outperforming Slim Fly’s ~89%. Crucially, PolarFly supports 50% more feasible configurations than Slim Fly, making it much easier to map onto standard commercial routers which typically use port counts in powers of two (e.g., 32, 64, 128).

Throughput and Cost

In simulations, PolarFly sustained higher saturation bandwidth than Dragonfly and Slim Fly across various traffic patterns. From a financial perspective, for a 1,024-node configuration, PolarFly offers a 5.19x cost reduction compared to traditional Fat Trees and is approximately 20% cheaper to build than Slim Fly.

Resilience

The topology exhibits remarkable fault tolerance. Even when 55% of the links fail, the network diameter only increases from 2 to 4, ensuring continued connectivity and functionality under extreme hardware stress.

Conclusion and Future Work

PolarFly successfully bridges the gap between theoretical graph theory and practical data center engineering. By providing a high-density, low-diameter topology that is both cost-effective and expandable, it offers a viable alternative to traditional hierarchical networks.

While the transition to diameter-3 during extreme expansion and the complexity of adaptive routing are noted limitations, the benefits in latency and cost are substantial. Future research may explore PolarFly's application in disaggregated memory systems, where the ultra-low latency of a diameter-2 network could provide even greater performance gains.

Summary updated

BibTeX
@inproceedings{lakhotia2022polarfly,
  author = {Kartik Lakhotia and Maciej Besta and Laura Monroe and Kelly Isham and Patrick Iff and Torsten Hoefler and Fabrizio Petrini},
  title = {{PolarFly: A Cost-Effective and Flexible Low-Diameter Topology}},
  booktitle = {SC},
  year = {2022},
  url = {https://openreview.net/forum?id=ISQyh3UPzX},
}
Figure