NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation

Momin Haider, Ming Yin, Menglei Zhang, Arpit Gupta, Jing Zhu, Yu-Xiang Wang

TL;DR

Reinforcement learning (RL) has transformed fields like robotics and gaming, but its application in computer networking has been bottlenecked by a lack of realistic simulation environments. To bridge this gap, researchers have introduced NetworkGym, a high-fidelity, open-source simulation framework. Alongside this "Simulation-as-a-Service" platform, the authors propose Pessimistic TD3 (PTD3), a novel offline RL algorithm that leverages value-function pessimism to handle the unique uncertainties of network traffic, significantly outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

While RL excels in standardized environments, networking involves intricate, multi-layered protocols (like 5G, LTE, and Wi-Fi) and highly dynamic traffic patterns that are notoriously difficult to simulate.

Specifically, this paper tackles multi-access traffic splitting: the challenge of dynamically deciding how to distribute data across multiple network connections simultaneously (e.g., sending some packets over Wi-Fi and others over a cellular network) to maximize throughput and minimize latency.

Architecture of the Generic Multi-Access (GMA) protocol for traffic splitting.

Existing RL benchmarks, such as D4RL, fail to capture these "real-world" networking nuances. This creates a "transferability gap" where algorithms that excel in standard benchmarks often fail when deployed in actual network simulations.

Methods

The authors contribute both a robust simulation platform and a novel RL algorithm:

  1. NetworkGym: A full-stack framework utilizing the industry-standard ns-3 simulator as its backend. It employs a "Simulation-as-a-Service" architecture, separating the RL Agent (typically written in Python) from the Environment (C++ based ns-3). This modularity allows for easy deployment, parallelization, and the integration of modern ML libraries without requiring researchers to rewrite complex networking protocols.
  2. GMA Protocol Integration: The environment natively simulates the Generic Multi-Access (GMA) protocol, an IETF-proposed standard designed for managing traffic across heterogeneous networks.
  3. Pessimistic TD3 (PTD3): An extension of the TD3+BC algorithm. Instead of relying solely on Behavioral Cloning (which forces the policy to stay close to the dataset), PTD3 uses a Fisher information matrix to estimate uncertainty in Q-values. It applies "pessimism" by penalizing actions where the model is uncertain, ensuring the learned policy is robust and safe for deployment in volatile network conditions.

Datasets

To test how well RL can learn from suboptimal but realistic data, the authors introduced three new offline datasets generated by different hand-crafted heuristic policies:

  • throughput_argmax: A simple policy favoring the channel with the highest link capacity, resulting in low data diversity.
  • system_default: A policy that adjusts traffic ratios based on predefined delay thresholds.
  • utility_logistic: A more complex heuristic balancing throughput and delay, yielding high data diversity and coverage.

Key Insights and Ideas

  • The "Normalization" Trap: A surprising finding was that standard state-normalization techniques—often used by default in offline RL libraries—actually hurt performance in networking environments. Disabling these defaults was necessary for several algorithms to finally beat the baseline.
  • Failure of Popular Algorithms: State-of-the-art algorithms like CQL (Conservative Q-Learning) frequently failed to outperform simple hand-crafted heuristics. This underscores that networking is a distinct challenge compared to the MuJoCo or Atari environments typically used in RL research.
  • Pessimism > Cloning: In networking, the "long tail" of data (such as rare congestion events) is critical. Theoretically motivated pessimism (PTD3) proved far more effective at handling these edge cases than simple behavioral cloning (TD3+BC).

Key Results

The empirical evaluations highlight the strength of the proposed PTD3 algorithm against a suite of baselines.

Performance comparison of offline RL algorithms across different datasets.
  • PTD3 Dominance: PTD3 achieved an average return of 0.946, significantly outperforming other offline RL methods like CQL (0.749) and IQL (0.848).
  • Beating the Experts: On the utility_logistic dataset, PTD3 achieved a score of 1.079, effectively "learning" to be better than the heuristic policy that generated its own training data (which scored 0.949).
  • Closing the Gap: While online algorithms (like PPO and SAC) still provide a "soft upper bound" (scoring around 1.1–1.2), PTD3 is the closest an offline algorithm has come to matching that performance without needing to interact with the live environment during training.

Limitations and Future Work

  • Fixed Scale: The current experiments use a fixed number of User Equipments (UEs). Adding or removing a device currently requires retraining the model from scratch.
  • Computational Cost: PTD3 requires calculating and storing a Fisher information matrix. For very large neural networks, the $d \times d$ memory requirement (where $d$ is the number of parameters) can become a bottleneck on standard GPUs.
  • Future Directions: The authors plan to explore more complex movement patterns (such as random walks) and investigate methods to featurize a variable number of users, making the models more flexible for real-world, dynamic deployment.

Summary updated

Abstract

Mobile devices such as smartphones, laptops, and tablets can often connect to multiple access networks (e.g., Wi-Fi, LTE, and 5G) simultaneously. Recent advancements facilitate seamless integration of these connections below the transport layer, enhancing the experience for apps that lack inherent multi-path support. This optimization hinges on dynamically determining the traffic distribution across networks for each device, a process referred to as multi-access traffic splitting. This paper introduces NetworkGym, a high-fidelity network environment simulator that facilitates generating multiple network traffic flows and multi-access traffic splitting. This simulator facilitates training and evaluating different RL-based solutions for the multi-access traffic splitting problem. Our initial explorations demonstrate that the majority of existing state-of-the-art offline RL algorithms (e.g. CQL) fail to outperform certain hand-crafted heuristic policies on average. This illustrates the urgent need to evaluate offline RL algorithms against a broader range of benchmarks, rather than relying solely on popular ones such as D4RL. We also propose an extension to the TD3+BC algorithm, named Pessimistic TD3 (PTD3), and demonstrate that it outperforms many state-of-the-art offline RL algorithms. PTD3's behavioral constraint mechanism, which relies on value-function pessimism, is theoretically motivated and relatively simple to implement. We open source our code and offline datasets at github.com/hmomin/networkgym.
BibTeX
@inproceedings{haider2024networkgym,
  author = {Momin Haider and Ming Yin and Menglei Zhang and Arpit Gupta and Jing Zhu and Yu-Xiang Wang},
  title = {{NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=3814z76JNM},
}
Figure