TL;DR
Cellular networks present unique challenges for traditional TCP congestion control, leading to high latency. This paper introduces DeepCC, a Deep Reinforcement Learning (DRL) based "plug-in" that enhances existing TCP schemes to significantly reduce queuing delay in cellular networks without sacrificing much throughput.
What Problem Does This Paper Address?
TCP congestion control (CC) algorithms are crucial for managing network traffic. However, traditional CC schemes, designed for wired networks, often struggle in the highly dynamic and variable environments of cellular networks. These networks suffer from issues like fluctuating channel quality, radio scheduling delays, and deep buffers, leading to significant "bufferbloat" – high queuing delays. This bufferbloat is particularly detrimental to emerging delay-sensitive applications like real-time gaming and virtual reality. While many new TCP schemes have been proposed, the authors question whether designing entirely new protocols is the most efficient approach. Instead, they aim to create a general framework that can improve the performance of existing TCP schemes, making them better suited for cellular environments.
Methods
The core of DeepCC is a Deep Reinforcement Learning (DRL) agent. This agent acts as a "plug-in" that sits on top of existing TCP implementations. Instead of replacing the TCP logic, DeepCC learns to dynamically adjust the maximum congestion window (cwnd_max) allowed for the underlying TCP.
Here's a breakdown of the key components:
- Monitor Block: This component collects real-time network statistics, such as average packet delay, delivery rate, and the current cwnd from the underlying TCP.
- State Generator Block: This module takes the collected statistics and the application's desired target delay to create an input state vector for the DRL agent.
- DRL Agent: This is the brain of DeepCC. It uses a Deep Deterministic Policy Gradient (DDPG) algorithm to learn a policy that maps the observed network state to an optimal cwnd_max. The agent aims to keep the average packet delay below the target while maximizing throughput.
- Reward Generator Block: This component provides feedback to the DRL agent, rewarding it for keeping delays low and throughput high, and penalizing it for exceeding the target delay.
- Filter Kernel: To handle the noisy and dynamic nature of cellular networks, DeepCC employs a filter kernel. This kernel preprocesses the state information, making the learning process more efficient and robust.
- Recurrent Structure: To address the lack of a perfect Markov property in the observed states, DeepCC uses a recurrent structure to incorporate a history of past observations into the current state.
The key novelty lies in DeepCC's approach as a plug-in that enhances existing TCPs, rather than a clean-slate replacement, and its application of DRL to this specific problem in cellular networks.
Datasets
The paper uses a combination of:
- Emulated LTE network environments: Trained using Mahimahi, a trace-based emulator, with over 100,000 variations of cellular link bandwidths.
- Real-world LTE cellular traces: Collected from T-Mobile, Verizon, and AT&T in New York City and New Jersey, covering stationary and various moving scenarios. These traces were used for both training and evaluation.
The authors do not introduce a new benchmark dataset but rather leverage existing real-world cellular traces to evaluate their approach.
Key Insights and Ideas
- Plug-in approach is viable: The authors demonstrate that it's not always necessary to design entirely new TCP protocols. A well-designed learning-based plug-in can significantly improve existing schemes.
- DRL excels in dynamic environments: DeepCC's success highlights the power of DRL in adapting to the complex and unpredictable nature of cellular networks, outperforming traditional heuristic-based TCPs.
- Delay vs. Throughput trade-off management: DeepCC effectively learns to balance the inherent trade-off between minimizing delay and maximizing throughput, a critical challenge in cellular networks.
- Generalization is achievable: The DRL agent trained on a specific set of cellular traces shows good performance on unseen traces and real-world scenarios, indicating effective generalization.
- Filter kernel significantly aids learning: The use of a filter kernel demonstrably speeds up the training process and improves the overall performance of DeepCC.
Key Results
- Significant Delay Reduction: DeepCC enhanced TCP schemes achieved substantial reductions in queuing delay. For example, DCubic achieved 300% lower queuing delay, and DBBR achieved 175% lower queuing delay compared to their base versions, with only a ~6% compromise in throughput.
- Outperforming State-of-the-Art: DeepCC-enhanced classic TCP schemes (like Illinois and Westwood) were shown to outperform state-of-the-art cellular-specific and clean-slate learning-based designs.
- Improved Utilization: In some scenarios, DeepCC-enhanced schemes achieved higher utilization (up to 10% higher than clean-slate learning designs).
- Robustness to Network Conditions: DeepCC demonstrated good performance across various scenarios, including different buffer sizes and varying mRTTs, and even when the bottleneck link switched between cellular and wired segments.
- Low Overhead: Despite using DRL, DeepCC exhibited lower CPU overhead compared to most other state-of-the-art TCP schemes.
Limitations and Future Work
- Target Delay Assumption: DeepCC's effectiveness is tied to the application's ability to specify a target delay. While flexible, this requires some level of application awareness.
- Kernel Implementation: The current DeepCC DRL agent is implemented in user-space, and a full kernel implementation is suggested for further performance gains.
- Generalization to Other Networks: While the framework is general, the current objective function is tailored for cellular networks. Adapting it to other network types is a future direction.
- Theoretical Guarantees: The paper acknowledges that providing absolute performance guarantees in dynamic networks like the internet is inherently difficult without network-level admission control.
- Fairness: While DeepCC generally maintains or improves fairness, further investigation into its impact on fairness across different scenarios is suggested.
- Optimization of DRL Agent: Further optimization of the DRL agent and its kernel implementation is a clear area for future work.
Summary updated