FNCC: Fast Notification Congestion Control in Data Center Networks

Jing Xu, Zhan Wang, Fan Yang, Ning Kang, Zhenlong Ma, Guojun Yuan, Guangming Tan, Ninghui SunICPP 2024 · arXiv

TL;DR

Data centers are struggling with network congestion due to rapidly increasing link speeds and limited buffer sizes. This paper introduces FNCC, a novel congestion control mechanism that significantly speeds up congestion notification by piggybacking network telemetry data onto ACK packets. This allows for much faster sender reactions, leading to reduced flow completion times and improved network utilization.

What Problem Does This Paper Address?

Modern data centers are the backbone of cloud computing, supporting a diverse range of applications from web search to machine learning. These applications demand both high throughput and ultra-low latency. To meet these demands, data center networks are constantly upgrading to faster links (up to 400Gbps and beyond). However, this rapid increase in link speed has outpaced the growth of switch buffer sizes, creating a critical bottleneck. When congestion occurs, existing congestion control mechanisms like DCQCN and HPCC are too slow to react. This delay, often taking a full Round-Trip Time (RTT) to notify the sender, leads to:

  • Deep Queues: Buffers fill up quickly, increasing latency.
  • PFC Pauses: Priority Flow Control (PFC) is triggered to prevent packet loss, which can lead to network deadlocks and performance degradation.
  • Sluggish Response: Senders are slow to reduce their transmission rates, exacerbating congestion.
  • Underutilization: Inefficient use of network resources when congestion clears.

The paper argues that a fast congestion control mechanism is urgently needed to address these issues, focusing on accurate congestion detection, fast notification, and precise rate adjustment.

Methods

FNCC (Fast Notification Congestion Control) builds upon the HPCC (High Precision Congestion Control) framework but introduces key innovations to accelerate the notification process.

  • Sub-RTT Notification via ACK Piggybacking: The core idea is to leverage the return path of ACK packets. Instead of waiting for data packets to carry In-Network Telemetry (INT) information back to the sender (as in HPCC), FNCC embeds INT data directly into the ACK packets. This allows congestion information from intermediate switches to reach the sender in less than one RTT.
  • Receiver-Informed Congested Flow Count: To further speed up the response to last-hop congestion (the most critical point), FNCC's receiver informs the sender about the number of concurrent congested flows. This allows the sender to directly calculate a fair rate for the congested flows, bypassing a step-by-step adjustment process.
  • Symmetric Routing and Path Consistency: The design relies on the assumption that data packets and their corresponding ACK packets follow the same path. This is achieved through mechanisms like symmetric routing tables and ECMP (Equal-Cost Multipath) load balancing, ensuring that INT data embedded in ACKs accurately reflects the congestion experienced by the data packets.
  • Last-Hop Congestion Speedup (LHCS): A specialized algorithm is proposed for last-hop congestion. When detected, it directly sets the sender's window to a calculated fair rate, enabling a very rapid response.

Datasets

The paper evaluates FNCC using simulations based on publicly available data center traffic traces, specifically mentioning:

  • WebSearch traffic (throughput-sensitive large flows)
  • FB_Hadoop traffic (latency-sensitive small flows)

The authors also mention using a typical dumbbell topology and a large-scale fat-tree topology (k=8 with 128 servers) for their experiments. No new benchmarks were introduced.

Key Insights and Ideas

  • The ACK Packet is a Fast Lane: The most significant insight is recognizing that ACK packets, by their nature, traverse the same network paths as data packets but return much faster. This makes them an ideal vehicle for timely congestion information.
  • Last-Hop Congestion is Critical: The paper highlights that last-hop congestion is particularly problematic and requires specialized, rapid handling. FNCC's approach of informing the sender about the number of congested flows directly addresses this.
  • Sub-RTT is Achievable and Beneficial: The experiments demonstrate that achieving sub-RTT notification is not only possible but leads to substantial improvements in performance metrics like flow completion time and queue depth.
  • Hardware Feasibility: The proposed FNCC mechanism is designed to be hardware-friendly, requiring minimal additional memory and computational resources on switches and hosts, making it practical for deployment on existing infrastructure.

Key Results

FNCC demonstrates significant improvements over existing congestion control mechanisms like HPCC and DCQCN:

  • Reduced Flow Completion Time (FCT): FNCC reduces FCT by 27.4% compared to HPCC and 88.9% compared to DCQCN for short flows (under 100KB) in Hadoop traffic. For longer flows (over 1MB) in WebSearch traffic, it reduces medium FCT by 12.4% vs. HPCC and 42.8% vs. DCQCN.
  • Lower Queue Depths: FNCC consistently maintains the shortest queue lengths at congestion points across various link rates (100Gbps, 200Gbps, 400Gbps). For example, it reduces queue depth by 37.5% in first-hop congestion and 29.5% in middle-hop congestion.
  • Fewer Pause Frames: FNCC generates significantly fewer PFC pause frames compared to HPCC and DCQCN, reducing the risk of deadlocks and network instability.
  • Higher Network Utilization: FNCC maintains high link utilization, especially when congestion clears, by promptly signaling senders to ramp up their rates.
  • Faster Response and Convergence: FNCC is the first to detect congestion and slow down, and it converges to fair rates more smoothly and quickly than HPCC and DCQCN.

Limitations and Future Work

  • Path Consistency Assumption: FNCC's effectiveness relies on data packets and ACK packets taking the same path. While the paper discusses methods to ensure this (symmetric routing, ECMP), complex network topologies or dynamic routing changes could potentially break this assumption.
  • Parameter Tuning: While FNCC aims to reduce complexity, like any congestion control mechanism, some parameters (e.g., the threshold a for last-hop detection) might require tuning for optimal performance in different environments.
  • Future Work: The paper doesn't explicitly detail future work, but the demonstrated success suggests potential avenues for further research:
  • Exploring FNCC's performance in more complex and dynamic network environments.
  • Investigating its interaction with other network protocols and features.
  • Developing adaptive parameter tuning mechanisms.
  • Extending FNCC to support different types of traffic and network architectures beyond data centers.

Summary updated

BibTeX
@inproceedings{xu2024fncc,
  author = {Jing Xu and Zhan Wang and Fan Yang and Ning Kang and Zhenlong Ma and Guojun Yuan and Guangming Tan and Ninghui Sun},
  title = {{FNCC: Fast Notification Congestion Control in Data Center Networks}},
  booktitle = {ICPP},
  year = {2024},
}
Figure