A congestion controller decides how fast each sender pushes data into the network. Deep reinforcement learning, where an agent learns a control policy by trial and error against a reward, can produce controllers that beat hand-designed ones in simulation. The trial-and-error part is the problem. While the agent is still bad, it overfills router buffers, drops packets, and starves other traffic. In a simulator this costs nothing. On a real link it is exactly the harm a network operator will not accept. We propose to handle this with a curriculum of safety interventions rather than a curriculum of easier tasks. Early in training, a classical protocol such as BBR or CUBIC takes over whenever the learner drives queueing delay past a threshold. An automated "teacher" then decides, across training rounds, how aggressive these takeovers should be and when to withdraw them. At the end, the learner runs alone.
The two-sentence version is this. Instead of making the network easier for a learning controller, we let a trusted classical controller step in whenever the learner is about to do damage, and we learn a schedule for removing that protection. The claim is that this schedule can cut training-time damage by at least an order of magnitude without hurting final quality, and without leaving the learner permanently dependent on the protection. The method comes from Safe Reinforcement Learning via Curriculum Induction (CISR). In that work, a teacher using Bayesian optimization scheduled reset controllers and cut Lunar Lander training failures from over 1,200 to fewer than 3, while keeping 88–92% final success. It has only been tested on toy control tasks. Networking is a natural home for it because classical protocols already make good, well-understood fallback controllers.
Why this gap, and why this version of it
Existing work that connects network learning with curricula varies task difficulty, not safety. FlagVNE is the closest explicit case. It uses meta-learning and advances virtual-network-embedding requests from 2 to 10 nodes once policy entropy falls below a threshold. Its curriculum is one-dimensional and applies to static graphs. It checks feasibility only after each decision, so nothing limits the damage exploration does during training. Reinforcement Learning for Datacenter Congestion Control trains on 2–8 flows and transfers zero-shot to 8,192. That is an implicit easy-to-hard progression, not a formal curriculum. The same paper motivates our work directly: PPO and REINFORCE suffered catastrophic packet loss in its fabric, and the authors escaped only by deriving an analytic gradient for a differentiable reward (ADPG). Aurora trains PPO with uniform randomization over 1.2–6 Mbps links and generalizes to 128 Mbps. That result constrains our proposal: it shows that a plain difficulty curriculum over link parameters may add little. We therefore deliberately do not propose one.
What makes training-time safety matter rather than remain academic is the gap between simulation and real paths. Pantheon found that even calibrated emulators miss real-path throughput and delay by about 17% on average. Any serious deployment will therefore want to fine-tune on real or production-like paths. That is where a learner's exploratory damage lands on real users. DeepCC makes a related point from the other direction. It keeps classical TCP in the loop and lets a learned agent cap the congestion window, and it reports 300% lower queuing delay for its CUBIC variant with about 6% throughput cost on LTE traces. DeepCC treats the classical protocol as permanent scaffolding. We treat it as scaffolding to be removed.
Core Claim
A teacher that schedules classical-controller takeovers, and gradually relaxes them, will produce a learned rate controller with three properties. First, during training it causes at least 10× fewer "failure" events than unconstrained PPO. We define a failure as a buffer-overflow drop burst, or queueing delay above a stated budget for longer than a stated window. Second, when evaluated alone it matches unconstrained PPO within noise on throughput, delay and fairness. Third, it outperforms a learner trained under a permanent takeover shield, because that learner never practiced recovering from its own mistakes.
Research Program
The student is an Aurora-style PPO rate controller. It observes a short history of latency gradient, latency ratio and sending ratio, and it is rewarded for throughput and penalized for latency and loss, as in the original formulation. An intervention is defined by two things: which fallback protocol is used (BBR, CUBIC, or a conservative fixed-rate backoff) and the queueing-delay threshold that triggers it. When the threshold is crossed, the fallback controls the sender for a fixed period and then returns control to the student. The student receives the resulting transition, as in CISR's reset-controller setup.
The teacher works in the same way as in CISR. Each "generation" trains a fresh student under a short sequence of interventions, such as a tight BBR takeover for the first third of training, a looser one next, and none at the end. It scores the student on held-out links with no intervention and a count of failures. It then uses GP-UCB, a Bayesian-optimization rule that trades off trying new schedules against reusing good ones, to choose the next generation's schedule. Teacher decisions happen only at episode boundaries. This matters because Syllabus measured 7.7–19.1% throughput loss for episodic curriculum updates against 59.0–88.7% for step-wise ones. We would integrate through that library's episodic interface.
Link conditions are sampled with uniform domain randomization over Pantheon's five-parameter space: bottleneck rate, propagation delay, queue size, loss rate, and delivery process. Because this is identical across all arms, any effect comes from the interventions and not from task ordering.
The teacher's cost is real: it trains several students. We report it two ways. One counts the teacher's full cost against our method. The other treats the teacher as amortized, meaning a schedule learned once on one family of links is reused for new students on a different family. The second claim is only credible if the transfer experiment below succeeds.
Baselines
The naive reference is the Aurora recipe: PPO trained with uniform link randomization and no protection. It fixes the unprotected failure count and the final quality to beat. It gets the same simulator, network size, environment-step budget, and hyperparameter search (the same number of trials over learning rate and entropy coefficient) as every learned arm.
The strongest safety competitor is constrained PPO with a Lagrangian penalty on the failure rate. This is the standard safe-RL answer, although the file cites no networking instance of it. It gets the same budget and a tuned constraint target. It is strong because it needs no fallback controller at all. Beating it means fewer training failures at equal final quality, or better quality at equal failures.
The strongest existing deployed-style method is DeepCC's plug-in design, which is permanent classical protection. We run it on the same Mahimahi LTE traces. The only reported numbers are relative to its own base protocols: 300% lower queuing delay for DCubic, 175% for DBBR, and about 6% throughput cost. No absolute delay or throughput numbers for DeepCC or Aurora on a shared benchmark are in the file. We therefore re-measure all arms ourselves rather than compare against published figures. BBR and CUBIC alone serve as the classical floor.
Two ablations isolate the key ingredient. A permanent-shield ablation keeps the tightest intervention for all of training, and we then evaluate the student with the shield removed. A hand-written schedule ablation relaxes the threshold linearly and drops it at a fixed point. If the hand-written schedule matches the learned teacher, we will say that the teacher is unnecessary and that the contribution is intervention-and-withdrawal itself. That result would still be useful and cheaper.
Why not just…?
Why not just train entirely in simulation, where failures are free? For a pure-simulation deployment this is the right answer, and our method would add nothing. The case for our method rests on real-path fine-tuning being needed, and Pantheon's 17% emulator error suggests it is. We test this directly. We pretrain in the Gym simulator, then fine-tune in calibrated Pantheon emulators treated as if they were live. We count failures only in the fine-tuning phase and check whether fine-tuning improves real-path metrics at all. If fine-tuning gives no gain, the motivation weakens, and we will report that.
Why not just keep a permanent safety shield, as DeepCC does? Permanent shields are safe and work well. However, the learner never experiences its own overload states, so it may fail once the shield is lifted. It also stays tied to the fallback's behavior and computational cost. The permanent-shield ablation measures this dependence. If shield-removed performance equals that of our withdrawn-schedule student, the withdrawal claim fails.
Why not just add a constraint penalty (Lagrangian RL)? Penalty methods learn to avoid violations only after committing them. CISR's unconstrained Lunar Lander baseline shows how many failures accumulate that way. This is a genuine threat when failures are cheap in number but costly per event, so the Lagrangian arm is the head-to-head comparison that decides the question.
Why not just use analytic gradients as ADPG does? ADPG removed PPO's packet loss in datacenters, but it needs a differentiable reward and model around a single bottleneck. Our intervention mechanism assumes only that a classical fallback exists, so it applies wherever such a fallback does. In a secondary experiment, we would test whether intervention scheduling lets plain PPO train in a many-flow fabric where PPO previously failed. This depends on having access to an equivalent simulator. The OMNeT++ NIC emulator used by ADPG may not be available.
Proposed Experiments
Training uses the Aurora Gym link simulator, with links drawn from Pantheon's five parameters. Evaluation uses Pantheon calibrated emulators and Mahimahi with the NYC/NJ LTE traces used by DeepCC. The main metrics are cumulative training failures and, on the final policy running alone, throughput, 95th-percentile queueing delay, loss, and Jain fairness against competing CUBIC flows. We use at least five seeds per arm and report confidence intervals.
The first decisive check is whether our method cuts failures by at least 10× relative to unconstrained PPO while final metrics stay within the seed-to-seed spread. The second is whether the student trained with withdrawal beats the permanent-shield student once both run without protection. The third is teacher transfer: a schedule learned on low-rate wired links is applied, without retuning, to LTE-trace training, and must keep most of its failure reduction. Failing the third check would mean the teacher's cost cannot be amortized. We would then recommend the hand-written schedule if it is competitive.
References
- Safe Reinforcement Learning via Curriculum Induction
- FlagVNE: A Flexible and Generalizable Reinforcement Learning Framework for Network Resource Allocation
- Reinforcement Learning for Datacenter Congestion Control
- A Deep Reinforcement Learning Perspective on Internet Congestion Control
- Pantheon: the training ground for Internet congestion-control research
- Wanna Make Your TCP Scheme Great for Cellular Networks? Let Machines Do It for You!
- Syllabus: Curriculum Learning Made Easy