Retreat or Escalate: Cascades That Can Stop at a Coarser Answer

Running several models on each input and combining their answers (an ensemble) makes a classifier more accurate, but it multiplies inference cost. Cascades reduce that cost: a cheap model runs first, and larger models run only when the cheap one is unsure. These cascades break down when there are many classes. With a thousand labels, even a cheap model that clearly sees "some kind of dog" spreads its probability over dozens of breeds. No single label crosses the confidence threshold, so nearly every input escalates to the expensive models. We propose a cascade that has three options at every stage instead of two. It can answer with a specific label, answer with a coarser node in the class hierarchy ("dog"), or pay for the next model. It chooses the option that buys the most specificity per unit of compute, under one guarantee on the error rate of the whole pipeline.

The hard part is not attaching a "climb the taxonomy" rule to each stage. The hard part is that a cascade is itself a hierarchy of irreversible decisions, so errors propagate through it. A cheap model that confidently commits to the wrong coarse node ends the pipeline, and no later model can repair it. Inputs that do escalate are a selected, harder population. A confidence threshold calibrated on all inputs is therefore miscalibrated for the inputs that actually reach stage two. Our claim is that two things fix this. The first is calibrating the entire retreat-or-escalate policy end to end. The second is triggering escalation by how much the next model is expected to deepen the answer, not by how confident the current model is about a leaf. Together these give risk-controlled cascades that keep their speedups at ImageNet scale, the regime where flat cascades are known to fail.

What Exists and What Is Missing

Efficient ensemble inference supplies the cascade machinery. IRENE treats ensembling as a halting problem, using a recurrent selector over a fixed model order. It reports a 56% inference-cost reduction on CIFAR-10 with slightly higher accuracy than the full ensemble. LazyStack adds two things: small learned aggregators ("substackers") that pool probabilities from every model run so far, and a learned, non-cheapest-first execution order. It reports a 2.5x speedup on CIFAR-100 while retaining 97% of ensemble accuracy. It also states that confidence-based exits lose their speedup on ImageNet-1K because probability mass is too diluted. That admitted failure motivates this proposal. Both systems use flat label spaces. Our method reuses LazyStack's prefix aggregation and differs in what the exit rule is allowed to output.

Hierarchical selective classification supplies the output space. Hierarchical Selective Classification starts at a single model's most likely leaf and climbs toward the root until the ancestor's probability mass exceeds a threshold. The threshold is calibrated by split conformal prediction so that the probability the true label is not under the returned node stays bounded. It reports a 14.94% hierarchical gain over flat selective classification on ImageNet-1K. Conformal prediction with constrained representation complexity does the analogous thing for prediction sets spanned by a few internal nodes. Both methods are post hoc on one fixed backbone that runs in full on every input, so neither saves compute. We reuse their climbing and calibration ideas, but place them inside a multi-model cascade where the calibration population changes from stage to stage.

Work on hierarchical error propagation constrains the design. Coarse-to-fine systems cannot undo a wrong coarse commitment: Brodley et al. show this for medical retrieval, Zhou and Mok for scan-then-zoom localization, and Castrejón et al. for generative cascades. One Rank at a Time proves that early errors amplify multiplicatively and argues for front-loading compute on early stages. Distilling Knowledge for Search-based Structured Prediction shows that a component trained only on clean inputs fails on the states that earlier mistakes produce. A cascade exit decision is exactly such an early, irreversible commitment. A model downstream of an exit gate sees only the residue of inputs that earlier stages declined to answer. HNE organizes ensemble members as a tree over shared trunks for anytime inference. Its tree is over parameters, not over classes, and it cannot answer at a coarser label.

Core Claim

Let $\pi$ be a cascade policy that, at each stage $t$, returns a taxonomy node $v$ or escalates. Let $R(\pi)$ be the probability that the true label is not a descendant of the returned node. Let $S(\pi)$ be the expected specificity, defined as $\log(L / L_v)$, where $L$ is the total number of leaves and $L_v$ is the number of leaves under $v$. Let $C(\pi)$ be the expected compute. We seek

$$\max_{\pi} \; S(\pi) - \lambda\, C(\pi) \quad \text{subject to} \quad R(\pi) \le \alpha,$$

with $\lambda$ the price of compute and $\alpha$ the user's tolerated hierarchical error rate.

The policy has two parts. First, at stage $t$, a climbing rule runs on the prefix-aggregated distribution from all models executed so far, using threshold $\theta_t$. This yields the deepest node the current evidence supports.

Second, a small depth-gain predictor estimates how much deeper the answer would get if stage $t+1$ ran, using the aggregated distribution and the current node's depth. Escalation happens when predicted depth gain times the price of specificity exceeds $\lambda$ times the next model's cost.

The thresholds $(\theta_1, \dots, \theta_T)$ form a low-dimensional vector. They are chosen jointly on held-out data by testing a finite grid of configurations for end-to-end risk at most $\alpha$, with a multiple-testing correction across the grid. The guarantee then covers the whole cascade, including the selection effect at each gate.

We make three predictions. First, flat cascades lose their speedup on ImageNet-1K, but retreat-or-escalate cascades do not, because ancestor mass is not diluted the way leaf mass is. Second, calibrating each stage on its own violates the end-to-end risk target or becomes overly conservative, and the gap grows with cascade depth. Third, most remaining errors come from confident wrong coarse commitments at stage one. Ordering policies that put a stronger model first should reduce these errors, the cascade analogue of front-loading.

Research Program

We build the cascade on ladders of pretrained classifiers of increasing cost over a shared label taxonomy. We use WordNet for ImageNet-1K, the 7-level taxonomy for iNaturalist 2021, the 20-superclass / 100-class split for CIFAR-100, and the botanical tree for PlantCLEF 2015. Following LazyStack, the prefix aggregators are small multilayer networks trained on the few execution trajectories that dominate traffic.

The depth-gain predictor is trained on the inputs that actually reach each stage, not on all inputs. This is a direct response to the exposure-bias finding of Liu et al.. We also compare against a version trained on all inputs, to measure how much this choice matters.

The analysis decomposes every final error by origin. A stage-one coarse commitment that was wrong is a propagated error. An error introduced by a later model is a late error. An error caused by a miscalibrated gate is a calibration error. This tells us whether propagation through gates is a real cost or a theoretical worry.

Baselines

The naive reference is the largest model alone, with the climbing rule of Hierarchical Selective Classification applied post hoc. It fixes the specificity attainable at full single-model cost. HSC's reported 14.94% hierarchical gain on ImageNet-1K is the reference for how much retreating helps a single model. The full ensemble with the same climbing rule gives the upper bound on specificity.

The strongest existing efficient method is LazyStack, run with its flat confidence exits and its learned ordering, with the climbing rule applied to whatever it finally outputs. It is the strongest competitor because it already has prefix aggregation and adaptive ordering. Its reported result is 2.5x speedup at 97% retained accuracy on CIFAR-100. For ImageNet-1K the file reports only that speedups shrink; there is no reported number on Lacuna for its ImageNet speedup, so we will measure it ourselves. IRENE is a second cascade baseline (56% cost reduction on CIFAR-10 as reported). HNE serves as an anytime baseline at matched FLOPs.

Every method gets the same model pool, the same calibration and test splits, and the same hyperparameter-search budget for its gates and aggregators. Every method is evaluated at the same target risk $\alpha$. Beating LazyStack means one of two things. Our method can reach the same specificity at a given $\alpha$ with clearly less compute on ImageNet-1K and iNaturalist. Or it can reach higher specificity at matched compute, while keeping the empirical end-to-end risk at or below $\alpha$.

Two ablations isolate the key ingredients. The first is "stage-wise HSC": each stage gets its own conformal threshold calibrated on all inputs, and the system escalates whenever the returned node is not a leaf. This removes both joint calibration and depth-gain escalation. The second keeps joint calibration but replaces depth-gain escalation with leaf-confidence escalation.

Why not just…?

Why not just run HSC on the cheap model and escalate whenever it does not reach a leaf? This is the most serious threat, and it is our first ablation. It ignores two things. Inputs reaching stage two are a selected population, so per-stage calibration need not deliver the end-to-end bound. And "not a leaf" is a poor signal for whether the next model will deepen the answer, since many inputs are inherently ambiguous below some depth. If this ablation meets the risk target and matches our specificity-compute curve, the contribution shrinks to an empirical finding that hierarchical exits rescue cascades at scale. We would report it that way.

Why not just use LazyStack with temperature-scaled confidence? LazyStack's own authors report that confidence-based exits stop saving compute on ImageNet-1K because leaf probabilities are diluted. Better calibration does not change the fact that no single leaf is likely. The head-to-head comparison on ImageNet-1K and iNaturalist, plotted against cascade depth, settles this.

Why not just distill or weight-average the ensemble into one model and apply HSC? SWA and subspace midpoints give ensemble-like accuracy at single-model cost. But compute stays fixed per input, and the subspace authors show that output-space ensembles still win when extra compute is available. Since IRENE finds that over 77% of CIFAR-100 inputs are solved by a single ResNet-18, a fixed-cost model overspends on easy inputs. We include an SWA model with HSC as a baseline. If it dominates our curve at every compute level, cascades are not worth their complexity in this setting.

Proposed Experiments

The main experiment plots specificity against average compute at target risks $\alpha \in \{0.05, 0.1, 0.2\}$ on ImageNet-1K, iNaturalist 2021, CIFAR-100, and PlantCLEF 2015, for every method above. The first decisive check is whether our cascade keeps a large speedup over the full ensemble on ImageNet-1K, where LazyStack's speedup is known to shrink.

The second decisive check is calibration. Over repeated random calibration/test splits, we compare stage-wise and joint calibration as the number of stages grows from two to five. If stage-wise calibration never violates $\alpha$ and is not more conservative, our joint-calibration claim fails.

The third check is the error decomposition, together with ordering policies that place a stronger model first. If propagated coarse errors are rare, front-loading is irrelevant here. If they dominate, the order of the cascade matters as much as its exit rule.