Threshold-Based Routing-Topology Co-Design for Optical Data Center

Peirui Cao, Shizhen Zhao, Dai Zhang, Zhuotao Liu, Mingwei Xu, Min Yee Teh, Yunzhuo Liu, Xinbing Wang, Chenghu Zhou

Introduction

Traditional data center networks (DCNs) are reaching a scaling limit. As bandwidth demands push toward 400Gbps and beyond, the electrical switches that form the backbone of these networks are becoming prohibitively expensive and power-hungry. Optical Circuit Switches (OCS) offer a promising alternative due to their high bandwidth and energy efficiency. However, OCS hardware suffers from significant reconfiguration latency—typically around 30ms—which is far too slow to react to microsecond-scale traffic bursts.

Existing solutions generally fall into two categories: traffic-aware designs that attempt to predict and rapidly reconfigure (often failing due to complexity) and traffic-agnostic designs that rotate through fixed patterns. The latter often incurs a heavy "bandwidth tax" by forcing traffic onto inefficient multi-hop paths via Valiant Load Balancing (VLB). This paper introduces TROD (Threshold Routing based Optical Datacenter), a novel architecture that leverages the "weak stability" of aggregate traffic to provide a high-performance, low-complexity optical DCN.

System Architecture

TROD is designed as a PoD-level (Point of Delivery) architecture. While individual server traffic is highly volatile, the aggregate traffic between PoDs—which may contain hundreds of servers—exhibits recurring patterns over longer timescales.

The TROD network architecture featuring an OCS-based Core Layer connecting multiple PoDs.

As shown in the architecture above, the Core Layer utilizes OCS to interconnect PoDs. TROD employs a Topology Engineering (ToE) approach where the optical layer is reconfigured infrequently (e.g., daily). Using a Progressive Filling Algorithm, TROD builds a topology that prioritizes direct links for heavy, recurring traffic pairs while maintaining a residual uniform mesh to handle unexpected bursts.

Threshold-Based Routing (tVLB)

The core innovation of TROD is its routing scheme, Threshold-based VLB (tVLB). Unlike traditional VLB, which spreads all traffic across indirect paths, tVLB prioritizes efficiency. Traffic is sent via the shortest direct path as long as the flow rate remains below a pre-defined threshold ($p$). Only the excess traffic from a burst is load-balanced across two-hop indirect paths.

The Multiple Flow Table (MFT) pipeline implementation of tVLB using standard switch features.

A key advantage of TROD is its compatibility with off-the-shelf hardware. As illustrated in the pipeline diagram, tVLB can be implemented on standard OpenFlow 1.3 or P4 switches (such as the Barefoot Tofino). It utilizes standard meter tables to monitor flow rates and DSCP remarking to distinguish between direct-path traffic and excess burst traffic that must be routed through the Group Table to indirect paths.

Key Results and Performance

The researchers evaluated TROD using real-world production traces from Facebook and synthetic bursty traces. The results demonstrate significant improvements over both traditional and state-of-the-art optical designs:

  • Flow Completion Time (FCT): TROD reduced FCT by 1.15–2.16× compared to Google’s Jupiter DCN and by at least 2× compared to other optical designs like Helios or Rotornet.
  • Comparison to Clos Networks: TROD outperformed the industry-standard non-blocking 1:1 Clos network when the optical layer was over-provisioned by a factor of just 1.2.
  • Implementation Accuracy: Physical hardware tests on a P4 switch confirmed that the tVLB rate-limiting mechanism is highly precise, with an error rate of less than 0.3%.

Conclusion

TROD demonstrates that by combining low-frequency topology reconfiguration with a clever threshold-based routing scheme, it is possible to overcome the reconfiguration latency of optical switches. By eliminating the "bandwidth tax" for the majority of traffic and utilizing standard hardware features, TROD provides a practical and efficient path forward for next-generation optical data centers.

Summary updated

Abstract

Despite the bandwidth scaling limit of electrical switching and the high cost of building Clos data center networks (DCNs), the adoption of optical DCNs is still limited. There are two reasons. First, existing optical DCN designs usually face high deployment complexity. Second, these designs are not full-optical and the performance benefit over the non-blocking Clos DCN is not clear. After exploring the design tradeoffs of the existing optical DCN designs, we propose TROD (Threshold Routing based Optical Datacenter), a low-complexity optical DCN with superior performance than other optical DCNs. There are two novel designs in TROD that contribute to its success. First, TROD performs robust topology optimization based on the recurring traffic patterns and thus does not need to react to every traffic change, which lowers deployment and management complexity. Second, TROD introduces tVLB (threshold-based Valiant Load Balance), which can avoid network congestion as much as possible even under unexpected traffic bursts. We conduct simulation based on both Facebook’s real DCN traces and our synthesized highly bursty DCN traces. TROD reduces flow completion time (FCT) by about 1.15- $2.16\times $ compared to Google’s Jupiter DCN, at least $2\times $ compared to other optical DCN designs, and about 2.4- $3.2\times $ compared to expander graph DCN. Compared with the non-blocking Clos, TROD reduces the hop count of the majority packets by one, and could even outperform the non-blocking Clos with proper bandwidth over-provision at the optical layer. Note that TROD can be built with commercially available hardware and does not require host modifications.
BibTeX
@article{cao2023threshold,
  author = {Peirui Cao and Shizhen Zhao and Dai Zhang and Zhuotao Liu and Mingwei Xu and Min Yee Teh and Yunzhuo Liu and Xinbing Wang and Chenghu Zhou},
  title = {{Threshold-Based Routing-Topology Co-Design for Optical Data Center}},
  journal = {IEEE/ACM Trans. Networking},
  year = {2023},
  doi = {10.1109/tnet.2023.3265276},
}
Figure