A common way to cut the cost of large models is a cascade. A small, cheap model answers every query first, and a decision rule (the "gate") sends only the hard-looking queries on to a bigger, more expensive model. On curated benchmarks this works very well. Confidence-based cascades over BERT sizes matched the largest model's accuracy on QQP using 11.07% of its compute (Model Cascading), and LLM cascades have reported API cost cuts of up to 98.3% (FrugalGPT). Every one of those numbers, however, was measured with the gate trained and tested on the same clean distribution. Deployment data is different. On the Camelyon17 hospital split of WILDS, accuracy falls from about 93% to about 70% when the hospital changes (WILDS). A gate that confidently passes shifted inputs through the small model, or that panics and escalates everything, can quietly erase either the accuracy or the savings.
This proposal makes a specific claim and tests it. Under real distribution shift, cascades fail mainly because the gate loses its ability to rank which queries the large model would actually fix. Its threshold drifting is the smaller part of the problem. We first build the measurement that separates these two failure causes on in-the-wild benchmarks. We then test a gate trained to predict escalation benefit across several training environments. Escalation benefit means whether the big model is right where the small one is wrong. The new gate also has a third option: stop early on "futile" inputs that neither model will get right. If the ranking claim is false, the cheap fix of re-setting the threshold on unlabeled deployment data will match our method, and the contribution reduces to a measurement study. The experiments are designed to tell these outcomes apart.
Core Claim
Write $s$ for the small model and $L$ for the large one. For an input $x$ with label $y$, define the escalation benefit
$$b(x) = \mathbb{1}[L(x)=y] - \mathbb{1}[s(x)=y] \in \{-1, 0, 1\}.$$
A cascade's cost-accuracy curve depends on two things. The first is how well the gate's score orders inputs by $b(x)$. The second is where its threshold sits. Existing gates use the small model's top probability (MaxProb), its distance from a uniform distribution (DTU), or a learned scorer. Each is a proxy for "the small model is wrong." That proxy drops a distinction that matters under shift. Many shifted inputs are wrong for both models, so $b=0$. Routing them to $L$ costs money and fixes nothing. This is the cascade version of the "futile sample" that Rethinking Calibration for Early-Exit Neural Networks identified inside single early-exit networks. We predict three things. On WILDS-style shifts, the area under the ROC curve for ranking $b(x)>0$ drops sharply for proxy gates. Re-setting thresholds without labels recovers the target compute budget but not the accuracy. A gate trained directly on $b$, with model selection by leaving out one training environment at a time, keeps much more of the in-distribution savings.
Relation to Prior Work
Three papers motivate this work by showing how good cascades and routers look in-distribution, while leaving shift untested. Model Cascading evaluates only GLUE/SuperGLUE-style tasks. Its own limitations section suggests out-of-distribution detection as future work. FrugalGPT trains its DistilBERT scorer on i.i.d. splits of the same dataset. RouterBench collects 405,467 outcomes across 11 LLMs and shows cascades stop paying off once the internal judge's error passes about 10%. It never asks what shift does to that error. We take RouterBench's cost-quality curve framing as our evaluation lens. We take the 10% figure as a concrete hypothesis: shift pushes gates past this point.
Two papers supply the method and also sharpen the contrast. Rethinking Calibration for Early-Exit Neural Networks and Failure Prediction Is a Better Performance Proxy for Early-Exit Networks Than Calibration show three things. Calibration error is a poor guide to early-exit efficiency. Ranking difficulty is what matters. A tiny two-layer meta-classifier for failure prediction, called EEFP, beats calibrated softmax thresholds. We adopt their ranking view and their small meta-classifier. We differ from them in three ways. We predict benefit from moving to a different model rather than failure of the current exit. We operate across heterogeneous models rather than exits of a single network. We evaluate on natural shifts in WILDS and DollarStreet, whereas their out-of-distribution test is DomainNet only.
A further set of results constrains the design. WILDS reports that IRM and CORAL failed to beat plain training on 8 of 10 datasets, and that out-of-distribution validation data is unavailable in practice. We therefore select models only on held-out training environments, never on test domains. Revisiting Knowledge Distillation under Distribution Shift found that a student's robustness is capped by its teacher's, and that simple logit distillation beats fancier feature distillation under shift. It studies static students only. Its "robustness ceiling" predicts that many shifted inputs will have $b=0$, which is exactly the futile mass our third gate option targets. Does Progress On Object Recognition Benchmarks Improve Generalization on Crowdsourced, Global Data? shows that bigger models widened the Europe–Africa accuracy gap from 5.4% to 19.1% on DollarStreet. Escalating to a larger model may therefore help some regions less than others, so we report savings per region. Certifying Capabilities from Finite Tests shows that certification cost grows with how little environments overlap. That result is a warning that no amount of in-distribution testing certifies a cascade's curve on a new hospital. Our leave-one-environment-out protocol is the empirical stand-in for that missing guarantee.
Research Program
The first part is a diagnostic. Take a cascade with a fixed gate and a fixed target budget on a new domain. We split its accuracy loss relative to in-distribution into two parts. The threshold part is what is recovered by re-setting the threshold to hit the same compute fraction on the new domain, which needs no labels. The ranking part is what remains, which we measure as the drop in the area under the ROC curve for predicting $b(x)>0$. We also report the fraction of compute spent on futile inputs, meaning those where $s$ and $L$ are both wrong. Finally, we report a savings-retention ratio: the compute saved on new domains divided by the compute saved in-distribution, at a fixed tolerated accuracy loss of one point relative to $L$ alone.
The second part is the benefit gate. It is a small two-layer network in the spirit of EEFP. Its inputs are the small model's top-$k$ probabilities and its disagreement across a few cheap test-time augmentations. Where available, it also uses the small model's penultimate-layer features. It outputs two heads: $\hat p(b>0)$, and $\hat p(\text{both wrong})$. The gate accepts the small answer, escalates to the large model, or stops with the small answer when the futile head fires. It is trained on pooled labeled examples from the WILDS training domains. These labels require running $L$ once offline. Hyperparameters are chosen by worst held-out-training-domain area under the cost-accuracy curve. If gains appear, the third part asks whether they come from the benefit target, from multi-environment training, or from the futile head.
Baselines
The naive reference points are running the large model on everything and running the small model on everything. These two fix the endpoints of every cost-accuracy curve. On Camelyon17 they must reproduce roughly the ~93% to ~70% drop reported by WILDS for plain training.
The primary proxy baselines are the MaxProb and DTU confidence cascades of Model Cascading. We run them with the same model sizes, thresholds swept over the same budget grid, and thresholds tuned on in-distribution validation data, as in the original paper. The strongest reported number for this family is matching the largest model at 11.07% of its compute on QQP. That figure is in-distribution. We found no reported result on Lacuna for any cascade or router on WILDS, DollarStreet, or GeoDE. The out-of-distribution numbers for these baselines will therefore come from our own runs.
The learned-gate baselines are a DistilBERT-style scorer as in FrugalGPT, and the KNN and MLP predictive routers of RouterBench. Each gets the same training domains and the same tuning budget as our gate.
The strongest existing method is EEFP from Rethinking Calibration for Early-Exit Neural Networks, adapted to predict small-model failure in the cascade. It uses the identical two-layer architecture, inputs, and training data as our gate. The only difference is its training target. That makes it a clean control. The file reports no DomainNet cost-accuracy figure for EEFP, so we will reproduce it.
The cheap-fix baseline re-sets each gate's threshold on unlabeled target-domain inputs so that it hits the target compute fraction. We count our method as winning only if it beats EEFP and the threshold-reset version of the best proxy gate on worst-domain area under the cost-accuracy curve, with non-overlapping seed intervals, on at least three of the five WILDS datasets.
Ablations isolate each ingredient. We train the gate on the failure target versus the benefit target. We train it on one pooled domain versus with environment-held-out selection. We run it with and without the futile head.
Why not just…?
Why not just run the large model on everything? This is the honest choice when budgets allow, and it is always one endpoint of our curves. But WILDS shows large, well-trained models still lose about 23 points on new hospitals. Richards et al. show that scaling widened regional gaps, so uniform large-model use buys less robustness than it costs. If our diagnostic finds that most shifted errors are futile ($b=0$), then the large model is mostly paying for nothing on exactly those inputs.
Why not just recalibrate the small model with temperature scaling? Temperature scaling preserves the ranking of inputs. If the failure is in the ranking, it cannot help. The early-exit papers (Kubaty et al., 2026; Kubaty et al., 2025) further show that well-calibrated models can give worse cost-accuracy trade-offs. We include temperature-scaled MaxProb anyway, because their evidence is in-distribution.
Why not just re-set the threshold on unlabeled deployment data? This is the real threat. It costs nothing and needs no labels. If threshold drift dominates, it will match our gate, and our diagnostic decomposition is built to reveal exactly that. In that case we would report that the gap is a budget-control problem, not a ranking problem, and drop the method claim.
Why not just train the small model with Group DRO or another robust objective? WILDS found such methods rarely beat plain training. Group DRO's gain on CivilComments from 56.0% to 70.0% worst-group accuracy relied on group labels. Still, a more robust small model could restore gate ranking. We therefore run the cascade with a Group DRO small model on CivilComments as an additional arm.
Proposed Experiments
For vision, we build cascades from ResNet and ViT ladders of increasing size on Camelyon17-WILDS, iWildCam2020-WILDS, and FMoW-WILDS. We also evaluate ImageNet-pretrained ladders on DollarStreet and GeoDE, with accuracy and savings broken out per region. For text, we use the BERT Mini-to-Large ladder of Varshney et al. on Amazon-WILDS and CivilComments-WILDS, reporting worst-group accuracy. We further extend to temporal shift with the divyam3897/TG splits, training gates on past years and testing on later ones. As an in-distribution sanity check, we reproduce the QQP result before trusting any out-of-distribution number.
The first decisive check concerns proxy gates on new domains. We test whether their AUROC for $b>0$ falls far enough that their implied judge error exceeds RouterBench's 10% viability line. If it does not, cascades are robust and the premise fails. The second decisive check concerns accuracy at matched compute after unlabeled threshold re-setting. We ask whether the benefit gate beats re-set EEFP on worst-domain accuracy at matched compute. If not, the method adds nothing beyond the diagnosis. The third check asks how much large-model compute the futile head saves on shifted domains. It must save a measurable fraction without lowering accuracy, since by construction those inputs were lost anyway. A further check asks whether escalation gains are uneven across DollarStreet regions. Uneven gains would mean that cascade savings, like benchmark accuracy, overstate deployment value unequally across populations.
References
- Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems
- FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Rethinking Calibration for Early-Exit Neural Networks
- MARS: A Benchmark for Multi-LLM Algorithmic Routing System
- Failure Prediction Is a Better Performance Proxy for Early-Exit Networks Than Calibration
- Revisiting Knowledge Distillation under Distribution Shift
- Does Progress On Object Recognition Benchmarks Improve Generalization on Crowdsourced, Global Data?
- Certifying Capabilities from Finite Tests: When Is It Possible?