A Deep Reinforcement Learning Perspective on Internet Congestion Control

Nathan Jay, Noga H. Rotman, Brighten Godfrey, Michael Schapira, Aviv TamarICML 2019 · OpenReview

Managing the flow of data across the internet is a high-stakes challenge. By moving away from rigid, handcrafted rules and toward learned policies, researchers demonstrate that AI can navigate the messy, unpredictable nature of modern networks more efficiently than traditional algorithms.

The Problem: The Internet's "Traffic Cop"

Internet Congestion Control (CC) determines how fast a sender (like a video streaming server) should transmit data to a receiver without overwhelming the network. The challenge is that the internet operates largely as a "black box." Senders do not know the exact capacity of the links or how many other users are competing for bandwidth.

Traditional protocols like TCP CUBIC rely on simple, hardcoded heuristics—for example, halving the sending rate if a packet is lost. However, these rules struggle with modern complexities. In mobile networks, packets might be lost due to poor signal rather than actual congestion. Handcrafting a single protocol that works perfectly across satellite, fiber, and cellular links is an uphill battle.

Enter Aurora: RL for Congestion Control

The authors reframe congestion control as a Reinforcement Learning (RL) problem, introducing Aurora, a deep RL-based congestion control protocol.

  • The Agent: The data sender.
  • The Action: Adjusting the sending rate (increasing or decreasing) at fixed monitor intervals.
  • The State: A history of three key metrics: latency gradient (is delay increasing?), latency ratio (current vs. minimum delay), and sending ratio (packets sent vs. acknowledged).
  • The Reward: A linear function rewarding high throughput while penalizing high latency and packet loss ($10 \cdot \text{throughput} - 1000 \cdot \text{latency} - 2000 \cdot \text{loss}$).

Using the Proximal Policy Optimization (PPO) algorithm, the researchers trained a surprisingly simple neural network (two hidden layers with 32 and 16 neurons). This simplicity suggests that network congestion patterns are identifiable without massive, billion-parameter models.

Key Insights and Training Dynamics

The Power of History Providing the agent with a short history of past network states was crucial. As shown below, a history length of $k=10$ allows the agent to recognize trends (e.g., "latency is trending upward") rather than just reacting to a single snapshot in time, leading to higher and more stable training rewards.

Training reward across different history lengths (k).

Distinguishing Loss Types Aurora learns to distinguish between "random loss" (e.g., signal interference) and "congestion loss" (full buffers). While traditional TCP slows down for any loss, Aurora learns that if latency isn't rising, the loss is likely random, allowing it to maintain a high sending rate.

Zero-Shot Generalization To train Aurora, the authors developed a custom OpenAI Gym environment simulating network links with randomized bandwidths (1.2 to 6 Mbps). Remarkably, when evaluated on industry-standard emulation tools like Mininet and Pantheon, Aurora successfully generalized to network environments far beyond its training data—including bandwidths up to 128 Mbps (over 20x its training limit).

Performance Results

In dynamic network tests, Aurora achieved throughput nearly matching BBR (a high-performance protocol from Google) but with significantly lower latency. It also achieved near-perfect link utilization across a wide range of capacities, far outperforming TCP CUBIC in high-loss environments.

Crucially, Aurora demonstrated exceptional adaptability. When bandwidth fluctuated wildly, Aurora tracked the "optimal" sending rate much more closely than traditional methods, which often over-reacted or under-reacted to changes.

Throughput comparison under fluctuating bandwidth conditions.

Limitations and Future Work

Despite these promising results, deploying RL for congestion control in the real world faces several hurdles:

  • Fairness: If an RL agent is only rewarded for its own performance, it might become "greedy" and starve other users (like those using standard TCP) of bandwidth. Training for "socially responsible" behavior is a major challenge.
  • Safety: In a production environment, a neural network making unexpected decisions could crash a network link. Providing formal safety guarantees for RL policies remains difficult.
  • Multi-Objective Optimization: Different applications have different needs (e.g., a video call requires low latency, while a large file download prioritizes high throughput). Future work involves creating models that can dynamically adapt their goals based on the application type.

Summary updated

Abstract

We present and investigate a novel and timely application domain for deep reinforcement learning (RL): Internet congestion control. Congestion control is the core networking task of modulating traf...
BibTeX
@inproceedings{jay2019deep,
  author = {Nathan Jay and Noga H. Rotman and Brighten Godfrey and Michael Schapira and Aviv Tamar},
  title = {{A Deep Reinforcement Learning Perspective on Internet Congestion Control}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=S1ZzEnWd-S},
}
Figure