Enough Evidence to Claim a Congestion Controller Works at the Tail

Proactive congestion controllers forecast traffic and reroute or reserve capacity before queues overflow. They are usually judged on averages: prediction error, mean throughput, average reliability. The failures operators care about are rare. A buffer overflows once in a hundred thousand scheduling frames, or a latency spike breaks a service agreement a few times a month. Rare outcomes run into the same problem as clinical trials in a mild flu season. If too few failure events are observed, a true difference between two controllers cannot be separated from noise, and any "significant" difference that does appear is likely exaggerated. We propose to measure how badly underpowered tail comparisons of network controllers are, and to fix this in simulation with rare-event estimation. The method copies ("splits") simulated network trajectories that climb toward overflow, so rare failures are sampled far more often. Controllers are compared on the same traffic randomness, and the result is a confidence interval whose coverage can be checked.

In short, the proposal has two parts. First, a standard power calculation shows that claims like "proactive beats reactive at 99.999% reliability" need months to years of observed traffic to settle by counting failures directly. Second, a paired, splitting-based simulation harness can deliver the same statistical answer with orders of magnitude fewer simulated frames. We then use it to re-test published proactive controllers where it matters, on overflow probability rather than averages.

Core Claim

The proactive congestion literature reports point averages with no power analysis. The LSTM forecaster in Predicting Bandwidth Utilization on Network Links Using Machine Learning reports RMSE 0.0122 for 15-second-ahead link utilization. NeuRoute reports under 0.05% error relative to the heuristic it imitates. Foresight backpressure in Backpressure on the Backbone reports roughly 100% average throughput gain. None of these numbers says how often each controller lets a queue overflow, or whether a difference in that rate is real.

The sample-size requirement follows from the usual normal approximation to comparing two small proportions. Detecting a difference between overflow rates $p_1$ and $p_2$ at 5% significance with 80% power needs about

$$n \approx \frac{(z_{0.975}+z_{0.8})^2\,(p_1+p_2)}{(p_1-p_2)^2}$$

observation windows per controller. Here $z$ are standard normal quantiles, and the approximation takes each binomial variance as roughly $p$ when $p$ is small. For $p_1 = 10^{-5}$ versus $p_2 = 2\times 10^{-5}$, this is about $2.4\times10^6$ windows per arm. With 1 ms scheduling frames that is under an hour of traffic. With the 15-second decision windows of link-level proactive control it is over a year.

Our claim has two parts. First, most published tail-relevant comparisons at link or path granularity are underpowered by this measure. Second, in trace-driven or model-based simulation, multilevel splitting with common random numbers produces valid confidence intervals on the overflow-probability difference at a small fraction of the brute-force cost. That makes tail comparisons, including formal non-inferiority tests for neural route emulators, routine rather than infeasible.

Relation to Prior Work

Two papers connect calibrated statistics to proactive network control, and both target the controller rather than its evaluation. Guaranteed Dynamic Scheduling of Ultra-Reliable Low-Latency Traffic via Conformal Prediction wraps a black-box traffic predictor in online conformal prediction, a feedback rule that nudges an allocation threshold after each success or failure so that the long-run failure rate matches a target. It restores 90% reliability when a biased predictor alone gets 82%, and lifts broadband utilization from 45% to 66% when the predictor is over-conservative. Its guarantee is asymptotic, it is tested only at a 90% target on a synthetic Markov traffic model, and its per-frame binary feedback carries almost no information at 1e-5 targets. MetaSTNet combines simulation-to-real meta-learning with cross-conformal intervals for hourly and daily cellular forecasts. Its coverage is marginal over the test set, and it never closes the loop to count overflow events. Both papers motivate this work: they show that calibrated guarantees are the right currency. Neither asks how much data is needed to confirm a guarantee holds at the tail, or to rank two controllers there.

The clinical studies in the file play a different role. They show concretely what underpowered comparisons look like. In HAI and NAI titer correlates, a mild season left the correlates analysis for the live attenuated vaccine inconclusive (p > 0.40). In the carbapenem septic-shock cohort, a doubled mortality rate still produced an odds ratio of 1.99 with interval [0.55; 5.34]. The Onyx versus coil study and the intracranial stenting study show that "no significant difference" is not evidence of non-inferiority. We borrow their statistical vocabulary: power, event counts, and non-inferiority margins. Networks differ in one respect that helps us. In simulation, a system state can be copied, so rare events can be manufactured rather than waited for. That is the lever medicine lacks.

Research Program

The first component is an audit. For each benchmark in the file, we compute the event counts implied by its reported setup and ask what tail difference it could have detected. The benchmarks are the iPerf/Netmate VirtualBox testbed, GÉANT2, the CAIDA-based European backbone with ITMGen traffic, and the synthetic Markov URLLC model. The output is a table of minimum detectable overflow-rate differences as a function of experiment duration and decision-window length.

The second component is the estimator. We define an importance function: a scalar "closeness to failure," such as the maximum normalized queue occupancy across bottleneck links. We choose intermediate levels $0 < b_1 < \dots < b_K = 1$, where $b_K$ is overflow. When a simulated trajectory first crosses $b_j$, we clone its full state, including the forecaster's hidden state and the controller's installed rules. The overflow probability is then the product of conditional crossing probabilities,

$$\hat P(\text{overflow}) = \prod_{j=1}^{K} \hat p_j, \qquad \hat p_j = \frac{\#\text{clones from level } b_{j-1} \text{ reaching } b_j}{\#\text{clones started at } b_{j-1}},$$

each of which is moderate in size and therefore cheap to estimate. The gain comes from cloning, not from re-labeling. On a single passive trace, this telescoping product equals the raw empirical frequency and yields no variance reduction. For this reason the method applies to simulators and emulators and not to live traffic, and we say so explicitly. To compare two controllers, we drive both from the same root traffic segments and the same random-number streams, which reduces the variance of the difference. Confidence intervals come from independent replications of the full splitting run.

The third component is inference built on the estimator. We run tests of whether a proactive controller has a lower overflow rate than a reactive one. We run non-inferiority tests of the form "the emulator's overflow rate is at most the exact heuristic's plus $\delta$," applied to NeuRoute-style imitators. We also run finite-window checks of whether the online conformal scheduler actually meets 99.9%, 99.99%, and 99.999% targets within one-hour windows, rather than only asymptotically.

Baselines

The naive reference point is brute-force Monte Carlo: simulate each controller for as long as the budget allows, count overflows, and report exact binomial (Clopper–Pearson) intervals and a two-proportion test. It is run on the same simulators with the same total count of simulated frames as the splitting method, including frames spent on clones. The proposal beats it if the interval on the overflow-rate difference is at least ten times narrower at equal compute, while empirical coverage stays at or above 95%.

The strongest existing evaluation practice is what the cited papers report: mean prediction error, mean throughput, and average reliability. The reference figures are LSTM RMSE 0.0122 from Predicting Bandwidth Utilization, under 0.05% fidelity error with 30 ms versus 120 ms decision time from NeuRoute, about 100% average throughput gain from Backpressure on the Backbone, and the 82%-to-90% reliability restoration from the conformal scheduler. No reported result was found on Lacuna for overflow probability at the 1e-4 or 1e-5 level for any of these systems. We will compute both the averages and the tail estimates on identical runs. The proposal matters if the two disagree in ranking or significance in a non-trivial fraction of controller pairs.

The strongest statistical competitor is tail extrapolation. One version fits a generalized Pareto distribution to queue peaks above a moderate threshold. The other fits the exponential decay of overflow probability in buffer size that queueing large-deviation theory predicts. Both use only a passive run. They get the same frame budget and threshold selection by the same held-out procedure. Splitting beats them if its intervals cover the ground truth where theirs do not, particularly under controllers that change dynamics near saturation.

Two ablations isolate the key ingredients. The first removes common random numbers, running splitting per controller independently, to isolate the pairing gain. The second replaces the queue-occupancy importance function with a naive one based on offered load, to show how sensitive the gain is to choosing the right "closeness to failure" signal.

Why not just…?

Why not just simulate longer? On the synthetic Markov model that is feasible, and we will do it to obtain ground truth. On GÉANT2 or the backbone simulator with 15-second windows, the power calculation above implies millions of windows per arm for a 1e-5 versus 2e-5 contrast, and every hyperparameter sweep multiplies that. Brute force remains the correctness check but cannot be the working method. The experiment reports the compute ratio directly.

Why not just use a frequent proxy like 99th-percentile latency or mean queue length? Proxies are cheap and well powered, and if they always ranked controllers the same way as overflow probability, this proposal would be unnecessary. Proactive controllers are exactly the kind that can trade average queueing for tail behavior, for example by pre-installing routes that help typical bursts and hurt flash crowds. We will measure rank agreement between proxies and splitting estimates across all controller pairs. High agreement would be a negative result we would report.

Why not just fit an extreme-value tail to a passive run? This is the real threat. It works on live traffic, where splitting cannot. Its weakness is the assumption that the tail shape seen at moderate levels continues to overflow. A controller that reacts near saturation breaks that continuity by design. The ground-truth experiments on the Markov model and on long GÉANT2 runs settle whether extrapolation is biased. If it is not biased, we will recommend it for live settings and reserve splitting for pre-deployment certification.

Why not just rely on online conformal guarantees? The guarantee in the conformal scheduler is long-run. Its finite-window deviation at 1e-5 targets is exactly what goes unmeasured. The guarantee is not an alternative to power-aware evaluation but a claim that needs it.

Proposed Experiments

We use four simulated environments. The synthetic Markov URLLC/eMBB model of the conformal scheduler admits numerical ground truth. The iPerf-driven SDN testbed setup of the bandwidth-prediction paper is reproduced in emulation with an LSTM forecaster. GÉANT2 hosts a NeuRoute-style imitator of a multicommodity-flow heuristic. The CAIDA/ITMGen backbone hosts reactive and foresight backpressure. Milano and Trento traces from MetaSTNet supply hourly demand envelopes, with bursts synthesized inside each hour. The clinical cohorts serve only as motivation and are not used as data.

The first decisive check is validity. On the Markov model, splitting intervals must cover the exact overflow probability at nominal rates across targets from 1e-3 to 1e-6. The second is efficiency: the ratio of frames needed to reach a fixed interval width, compared against brute force and extreme-value extrapolation. The third is the scientific payoff. We test whether foresight backpressure's average throughput gain comes with a statistically detectable reduction in overflow probability. We test whether a NeuRoute-style emulator passes a non-inferiority test against its heuristic at a margin $\delta$ of 1e-5. We test whether the conformal scheduler meets 99.99% and 99.999% within one-hour windows. The result of the third check that would most change practice is a documented case in which an average-metric winner is a tail loser, or in which a "comparable" emulator fails non-inferiority once the test has power.