When Does Risk-Controlled Throughput Forecasting Actually Control Risk? A Reproduction Study on LEO Satellite Link

Mayan Sharma, Kriti Saini, Devansh Behl, Suganya R, S. VijayaragavanarXiv 2026 · arXiv

TL;DR

This reproduction study reveals that state-of-the-art risk-controlled forecasting for LEO satellite links (BG-CFQS) works well in idealized lab settings but struggles with the temporal drift of real-world deployment. The authors demonstrate that while static calibration fails to meet tight risk budgets, online recalibration provides a robust remedy, tracking user-specified risk budgets within 1% and significantly improving detection performance.

What Problem Does This Paper Address?

Low Earth Orbit (LEO) satellite links, like Starlink, are notoriously volatile. Bandwidth capacity can swing by a factor of three within a single hour due to frequent handovers and 15-second scheduling cycles. For network operators, "admission control"—deciding how many sessions to allow—is a high-stakes gamble.

If you rely on a standard point forecast and it’s too optimistic, you over-promise capacity, leading to dropped sessions and violated SLAs. To solve this, researchers use risk-budgeted forecasting: instead of a single number, the system generates a conservative bound. The goal is to ensure that the actual capacity falls below this bound only as often as a user-defined "risk budget" (e.g., 5% of the time) allows. This paper investigates whether current methods actually deliver on these safety guarantees when faced with real-world conditions.

Methods

The authors perform a rigorous reproduction of BG-CFQS, the current state-of-the-art method which uses XGBoost with a pinball loss function and conformal prediction to select optimal quantiles.

Beyond simple reproduction, they test the method's sensitivity to data splitting (random vs. temporal) and evaluate two alternative mechanisms:

  1. Online Recalibration (ACI): Dynamically updating the operating point based on observed outcomes rather than treating it as a static constant.
  2. Group-Conditional Calibration (GCACI): Maintaining separate operating points for different "regimes" (e.g., grouping by satellite elevation or visible satellite count).

Datasets

The study utilizes four distinct datasets:

  • StarNet Traces: Three high-resolution (1 Hz) traces covering Chicago, Osnabruck, and Victoria, including satellite telemetry (elevation, distance, etc.) and weather data.
  • WetLinks: An independent, 180-day measurement campaign from Osnabruck using different instruments and no satellite telemetry, providing a "blind" test for the methods.

Key Insights and Ideas

  • The "Exchangeability" Trap: The original paper's success relied on random data splits. In a real deployment, a terminal faces a contiguous temporal stream. When the authors switched to a temporal split, the "risk-pass" rate plummeted from 3/3 datasets to just 1/3. Static guarantees evaporate when faced with temporal drift.
  • The 0.15 Risk Floor: A structural limitation was uncovered: BG-CFQS cannot serve a risk budget below 0.15. Because its candidate quantile set is bounded, the selection interval collapses at tight budgets, causing the system to overshoot the risk by up to 3.7x.

![Fig.

  1. BG-CFQS achieved OverRate against the requested budget. The "floor" prevents the method from honoring tight risk budgets.](/figures/art_561854eec74043dfacb520e5da3646ef)
  2. Level vs. Residual: A common intuition is that conditioning on satellite geometry (like elevation) should help. The authors found the opposite: while geometry predicts the throughput level, it carries almost no information about the forecaster's uncertainty (the residuals). Conditioning on these "noisy" covariates actually degraded performance.
  3. Ranking vs. Operating Points: Static calibration is essentially adding a constant to a forecast. This might shift your "operating point" on a curve, but it cannot change the order of predictions. Therefore, static calibration cannot improve ranking metrics like AUPRC; only online calibration, which applies different corrections at different times, can "improve the curve."

Key Results

  • Online Superiority: Online recalibration tracked the requested risk budget within 0.8% across all locations, whereas static calibration was consistently 16% to 21% off the mark.

![Fig.

  1. Online recalibration (blue) tracks the target line perfectly, while static calibration (yellow) consistently undershoots the risk budget.](/figures/art_dd9b2df643324843b85165acba6e2e78)
  2. Performance Gains: For latency spike detection, online calibration improved the Area Under the Precision-Recall Curve (AUPRC) by 43% to 51% on two out of three links compared to point forecasts or static bounds.
  3. Economic Impact: The authors defined a "break-even ratio" for risk control. They found that for enterprise and URLLC (Ultra-Reliable Low-Latency Communications) services, where a violated session-hour is ~4x more expensive than a sold one, these risk-controlled policies pay for themselves.

Limitations and Future Work

  • Small Sample Size: While the results are consistent, they rest on only four datasets.
  • Empirical Nature: The online layer’s budget control is empirical and lacks the finite-sample guarantees found in some theoretical frameworks.
  • Feedback Requirements: Online methods require immediate outcome feedback after each decision, which may not always be available in every networking hardware configuration.
  • The "Conditioning" Mystery: The authors failed to find a pre-test statistic that could reliably predict when group conditioning would help versus hurt, suggesting a need for more robust "stability tests" in future research.

Summary updated

Abstract

Risk-budgeted throughput forecasting converts a point forecast of LEO satellite capacity into a conservative bound whose overestimation rate is held at a user-specified budget, so that admission control can act on it safely. We reproduce the state of the art, BG-CFQS, on the three Starlink traces its authors used and on an independent fourth dataset, and report four findings. First, its risk guarantee is reproducible but conditional on the calibration split: under a random split we recover every published quantity, including risk-pass on 3 of 3 datasets, but under a contiguous temporal split, which is what a deployed terminal faces, it holds on 1 of 3. Second, independent of any split, the method cannot serve a risk budget below 0.15, the low end of its candidate quantile set. On all four datasets the achieved rate is identical at budgets of 0.05, 0.10 and 0.15, overshooting by 1.4x to 3.7x at the tightest. Third, static conformal calibration cannot improve a ranking, only shift an operating point; we confirm this exactly on latency spike detection, where online calibration improves AUPRC by up to 51%. Fourth, group-conditional online recalibration splits into two mechanisms. The online part replicates: it tracks the budget within 1% from 0.05 to 0.35, where static calibration is 16 to 21% off. The group-conditional part does not: on three of four datasets, conditioning on any available covariate, including measured serving-satellite geometry, is worse than not conditioning, and our attempt to predict which datasets benefit failed. What is new here is empirical, not algorithmic: we claim the findings, not the methods.
BibTeX
@misc{sharma2026when,
  author = {Mayan Sharma and Kriti Saini and Devansh Behl and Suganya R and S. Vijayaragavan},
  title = {{When Does Risk-Controlled Throughput Forecasting Actually Control Risk? A Reproduction Study on LEO Satellite Link}},
  year = {2026},
  eprint = {2611.06307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06307},
}
Figure