A model cascade sends each request to a cheap model first and passes it to an expensive model only when a rule decides the cheap answer is not good enough. Every published deferral rule is fitted once, on a fixed calibration set, and then frozen. Production traffic does not hold still: the mix of math, code, and chit-chat shifts, and the cheap model's confidence stops meaning what it meant during calibration. The obvious fix is to re-estimate the rule online. That fix runs into a problem that adaptive control (the study of controllers that estimate unknown, changing plant parameters while running) has known about for decades. A cascade learns whether the cheap model was wrong only on the requests it chose to defer, because only those get a reference answer from the expensive model. When the rule trusts the cheap model, the estimator receives no information. Errors in exactly the region the rule trusts become invisible.
This note proposes to treat the cascade's "how much does deferring this request help?" function as an unknown, time-varying plant parameter. We track it with a recursive estimator that has a forgetting factor, and we deliberately spend a small, designed fraction of the budget on probe deferrals. Probes are requests sent to the expensive model even though the rule would not send them, chosen so the estimator stays informed. In adaptive-control terms, the probes supply persistency of excitation. The claim to test is that, at equal realized spend, this estimator-plus-probing loop recovers accuracy after a distribution shift faster and more completely than three alternatives: frozen calibration, periodic recalibration on whatever was deferred, and simple budget-pacing controllers that only enforce a target spend.
Core Claim
The closest systems each cover part of the problem. C3PO calibrates exit thresholds once, from fewer than 100 unlabeled prompts, using agreement with the most powerful model as a label-free signal. Its cost guarantee assumes calibration and test data are exchangeable. When Does Confidence-Based Cascade Deferral Suffice? proves that the Bayes-optimal rule defers when the large model's probability of being correct exceeds the small model's by more than the deferral cost, not when the small model's own confidence is low. It shows that raw confidence fails under distribution shift and label noise. Its fix, a post-hoc predictor of this "headroom," is trained offline. Inter-Cascade adds online memory by caching reasoning strategies from the strong model. It does not adapt the deferral rule itself, and it does not address the fact that it only learns from deferred queries.
Our proposal differs in three ways. It estimates the headroom function online. It explicitly corrects for the censored feedback the cascade creates. It designs the probing needed to keep the estimate identifiable, which here means recoverable from the data the cascade actually sees. The technique comes from recursive least-squares contact estimation and online Kalman-filter identification in mobile manipulation. Both track drifting physical parameters from prediction errors. The second also shows that explicit identification becomes fragile when the signal is noisy. That finding constrains our design and motivates the ablations below.
Research Program
At step $t$ a request $x_t$ arrives with cheap features $z_t$. These are the small model's confidence, prompt length, a domain tag from a tiny classifier, and an embedding-cluster index. None of them requires an extra model pass. We model the probability that the large model would change the answer to a correct one as $g_{\theta_t}(z_t)=\sigma(\theta_t^\top z_t)$, where $\sigma$ is the logistic function and $\theta_t$ drifts over time. The cascade defers when $g_{\hat\theta_t}(z_t) > \lambda_t$. Here $\lambda_t$ is a price on expensive-model spend, adjusted by a slow budget-tracking update that is itself one of the baselines.
Every deferred request yields a label $y_t$: whether the two models disagree and the large one matches ground truth or, in the label-free variant, whether they simply disagree, following C3PO's self-supervision. With probability $\epsilon_t(z_t)$ we also defer a request the rule would have kept. We then update $\hat\theta$ by a forgetting-factor recursive logistic estimator, an extended Kalman filter step. Each observation is weighted by the inverse of its deferral probability, so trusted regions are not underweighted.
The probe rate is not uniform. It is set to keep the smallest eigenvalue of the estimator's information matrix above a floor, which concentrates probes in the confidently-kept region where information is scarce. The standard forgetting-factor analysis motivates a specific tradeoff. With forgetting factor $\gamma\in(0,1)$, drift rate $\rho$ (change in $\theta$ per step), observation noise variance $\sigma^2$, and excitation level $\underline{\kappa}$ (the information floor), the steady tracking error behaves roughly like
$$ \mathbb{E}\|\hat\theta_t-\theta_t\|^2 \;\lesssim\; \frac{1}{\underline{\kappa}}\Big(\sigma^2(1-\gamma) + \frac{\rho^2}{1-\gamma}\Big). $$
The first term is noise that is averaged over an effective window of $1/(1-\gamma)$ steps. The second is lag behind the drift. Without probes, $\underline{\kappa}$ collapses toward zero in trusted regions and the bound blows up. This is the cascade analogue of parameter "bursting" in under-excited adaptive loops.
We will derive the probe allocation that minimizes total cost. Total cost here is probe spend plus the accuracy lost to the residual error in the bound above. We will then measure how closely the empirical behavior follows it.
Baselines
The naive reference points are always-small and always-large. Together they define the accuracy and cost range on each stream.
The strongest existing method is C3PO's offline conformal threshold, calibrated on the first segment of each stream and then frozen. On stationary data it is very strong. C3PO reports 62.5% accuracy at \$0.0019 per question on MATH-500, against 57% at \$0.0053 for the most powerful model with self-consistency. It reports near-MPM accuracy at under 20% of MPM cost across 16 benchmarks, and only one cost-constraint violation in 300 runs. RouteLLM's matrix-factorization router, which reports a 50% MT-Bench performance gain while calling the strong model about 13% of the time, will be run with its released weights at a threshold fitted to the same budget. Both baselines get the same calibration prompts, the same model pair, and the same realized-spend target. Their tuning budget is a grid search on the first segment, equal in size to our hyperparameter search over $\gamma$ and the excitation floor.
The offline headroom predictor (Diff-01) from When Does Confidence-Based Cascade Deferral Suffice? is trained on the first segment. It isolates the value of the headroom target from the value of tracking it online.
Two adaptive competitors use the same features and spend. The first is sliding-window recalibration: C3PO refitted every $W$ requests on deferred data only. The second is a budget-pacing integral controller that adjusts a raw-confidence threshold to hit the target spend.
Two ablations isolate the key ingredients. One removes the probes; the other replaces designed probes with uniform random probes at the same rate. A third ablation removes forgetting.
Beating the baselines means two things. First, higher accuracy than every adaptive competitor at matched realized cost, averaged over shift segments, with non-overlapping bootstrap intervals across stream seeds. Second, a shorter recovery time after abrupt shifts. Recovery time is the number of requests until accuracy returns within one point of the per-segment hindsight-optimal threshold. On stationary segments, we must match frozen C3PO to within one point while spending the probe overhead.
Why not just…?
Why not just recalibrate C3PO on a sliding window? This is the most natural fix, and it is the reason the probe ablation exists. Recalibrating on deferred requests inherits the selection bias described above: the window contains almost no evidence about the region the rule trusts, so a shift that makes confidently-kept answers wrong goes undetected. If sliding-window recalibration with a small random holdout matches our method, then the contribution shrinks to the probe design and the forgetting analysis. We would report that plainly. The comparison against uniform probes at equal rate settles this question.
Why not just run an integral controller that pins spend to the budget? Budget pacing guarantees the spend but not the allocation. When the stream gets easier, it keeps deferring at the same rate and wastes money on requests the small model already handles. When confidence miscalibrates under noise, it defers the ambiguous items that the "Noise Sink" analysis shows the large model cannot fix either. This is a real threat on streams where only difficulty drifts and calibration holds. We include exactly such a stream, and we expect the gap there to be small.
Why not just add Inter-Cascade's strategy memory? Memory raises small-model accuracy, by up to 33.06% on reasoning tasks in Inter-Cascade, but it changes the plant without re-estimating the rule. After memory injection the small model's confidence means something new, so its frozen threshold drifts out of calibration. We run Inter-Cascade with and without our estimator on top. We expect the two to complement each other rather than compete.
Why not just tune the small model's confidence once, Gatekeeper-style? Gatekeeper improves deferral 7x to 10x on ARC by teaching the small model to be unconfident when wrong. That calibration is fit to the training distribution, and the shift results above say it will degrade. We treat a Gatekeeper-tuned small model as a stronger plant to start from, and we test whether online tracking still adds value on top of it.
Proposed Experiments
Streams are built from the file's benchmarks with scheduled drift: GSM8K, MATH-500, MMLU, ARC, CommonSenseQA, AQuA, and MedMCQA. We use three schedule types: abrupt domain switches (for example GSM8K to MedMCQA), gradual mixture drift, and label-noise injection. The noise injection reproduces the Noise Sink. The model pairs are an 8B and a 70B open model, plus an API small/large pair. Each stream is replayed over ten seeds.
We report four metrics: accuracy versus realized cost, cumulative budget overshoot, regret against the per-segment hindsight-optimal rule, and recovery time. We also report the label-free variant, which uses only agreement with the large model, to test whether C3PO's self-supervision survives online use.
The decisive check is the selection-bias test. Introduce a shift that only corrupts high-confidence small-model answers. The no-probe and sliding-window variants should fail to detect it, and the designed-probe estimator should recover at a probe cost predictable from the tracking bound. A secondary check sweeps $\gamma$ against the drift rate and tests whether the error-optimal forgetting factor follows the square-root drift relationship the bound predicts.
Multi-step agent trajectories, where AgentRouter reports a 6–9% accuracy drop on an unseen framework, are a natural follow-up. They are out of scope here unless trajectory data can be obtained.
References
- C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
- When Does Confidence-Based Cascade Deferral Suffice?
- Not only a helper, but also a teacher: Interactive LLM Cascade
- Stable Contact Guaranteeing Motion/Force Control for an Aerial Manipulator on an Arbitrarily Tilted Surface
- Model Predictive Robot-Environment Interaction Control for Mobile Manipulation Tasks
- RouteLLM: Learning to Route LLMs from Preference Data
- Gatekeeper: Improving Model Cascades Through Confidence Tuning
- AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows