Does More Data Cure Cascading Errors? Measuring the Cascade Penalty in Hierarchical Classifiers

Many classifiers make decisions in stages. They first pick a coarse category, such as an insect's order or a news story's broad topic, and then pick a fine category within it. Any mistake at the coarse stage is final, because the fine stage only ever sees the branch it was handed. Separately, a large body of evidence says that adding training data steadily lowers error, often faster than changing the algorithm does. Nobody has measured how these two facts interact. As data grows, does the extra error caused by staged decisions shrink toward zero, or does it level off at a floor that no amount of data removes?

This proposal measures that extra error directly. We call it the cascade penalty: how much worse a staged classifier does than a model that predicts the fine label from all the evidence at once. We trace it as a learning curve over one to two orders of magnitude of training data, on a synthetic hierarchy where the theory is known and on real taxonomies (BIOSCAN-1M insects, plus news and scientific-abstract topic hierarchies). The central claim is that the penalty comes almost entirely from rare fine classes that have not yet been seen enough times to be routed correctly. If that is right, the penalty should shrink as a predictable power law tied to how skewed the class frequencies are, and its value at full scale should be predictable from small subsets. The second claim is that error-correcting designs, such as predicting fine labels first or training later stages to distrust earlier ones, help mainly at small scale, and their advantage should shrink along the same curve. Where the penalty instead flattens to a floor, we expect the cause to be ambiguity in the coarse decision itself (noise, blur), and the method separates the two cases.

Core Claim

Let $E_{\text{pipe}}(P)$ be the fine-level error of a staged top-down classifier trained on $P$ examples, and let $E_{\text{joint}}(P)$ be the fine-level error of a model with the same backbone and budget that is never forced to commit to a coarse decision. The cascade penalty is

$$\Delta(P) = E_{\text{pipe}}(P) - E_{\text{joint}}(P).$$

Suppose a leaf class becomes reliably routable once it has been seen about $n^*$ times, and leaf frequencies follow a Zipf law $f_k \propto k^{-(1+a)}$, where $k$ is the frequency rank and $a$ sets how heavy the tail is. Then the probability mass of leaves still below the threshold at scale $P$ is $\sum_{k:\,P f_k < n^*} f_k \propto (P/n^*)^{-a/(1+a)}$. This gives the prediction

$$\Delta(P) \approx c\,(P/n^*)^{-a/(1+a)} + \Delta_\infty .$$

Here $c$ is the fraction of under-sampled leaves that actually get misrouted, and $\Delta_\infty$ is a floor caused by coarse ambiguity that does not depend on frequency. The exponent comes from the same counting argument that underlies the root-classification learning curve in Learning curves theory of hierarchically compositional data with power-law distributed features. That paper derives $\varepsilon(P)\sim P^{-a/(1+a)}$ for a single classifier that does not decompose the decision into stages.

What is new is a stated, falsifiable decomposition of the penalty into three parts: a tail term with a known exponent, a per-frequency-bin estimate of $n^*$ measurable on small subsets, and a floor $\Delta_\infty$. These parts tell a practitioner, before training at full scale, whether a staged architecture or a specific error-correcting mechanism is worth its cost at the target data size.

Relation to Prior Work

The theory work motivates the exponent but does not test it. Cagnetta and Kang study the Random Hierarchy Model, a synthetic grammar that generates sequences from tree-structured rules. They show that skewed rule frequencies turn step-like learning curves into smooth power laws, but their classifier makes one decision, not a sequence of stages. We ask whether staging changes the exponent or only the constant. When Does Bottom-up Beat Top-down in Hierarchical Community Detection? proves, for graphs, that top-down splitting fails earlier than bottom-up merging because early splits irreversibly corrupt later ones. It gives us a theoretical reason to expect a nonzero $\Delta$ at small scale, but it studies sparsity rather than training-set size.

Three empirical papers supply the methods we compare but leave the scaling question open:

We reuse all three as treatments and ask how their benefit scales with data.

Two results constrain the design. Hierarchical Text Classification (HTC) vs. eXtreme Multilabel Classification (XML): Two Sides of the Same Medal finds that trees built by clustering labels from data beat hand-curated topic trees on NYT. This seems to contradict Ahmed et al., where the curated taxonomy helps under noise. Our decomposition offers a reconciliation to be tested: curated trees should matter when $\Delta_\infty$ is large (noisy inputs), and data-driven trees when the tail term dominates. We therefore include both tree types.

Towards a Science of Scaling Agent Systems reports that stronger underlying models do not remove error amplification in sequential multi-agent setups. That warns us that a capacity increase alone may not move $\Delta_\infty$, which is why capacity is a separate axis in our design.

Research Program

Synthetic hierarchy. The first component is a controlled testbed: a taxonomic variant of the Random Hierarchy Model. Leaf classes are root symbols. Coarse classes are groups of root symbols that share production rules in the upper levels of the grammar, so the coarse label is recoverable from shallower correlations than the fine label. We sweep three things:

  • the Zipf exponent $a$ on leaf frequencies, which should move the tail exponent;
  • a coarse-label ambiguity knob that mixes production rules across coarse groups, which should create $\Delta_\infty$ and leave the exponent unchanged;
  • model width, which tests the capacity explanation.

Training sizes span about three orders of magnitude, with a convolutional network and a Transformer as backbones. Because the frequencies are known exactly, this testbed gives a clean test of whether the fitted exponent of $\Delta(P)$ matches $a/(1+a)$.

BIOSCAN-1M. The second component moves to the BIOSCAN-1M insect dataset, using its standard 50k, 200k and full 903,536-image training sets and the fixed 224,777-image test set. The hierarchy runs from order through family and genus to the finest level. Only 7.5% of specimens have expert species labels, so the finest level is the Barcode Index Number (BIN, a DNA-derived proxy for species). We report it separately on the expert-labeled subset, so that label sparsity is not confused with cascading.

Text hierarchies. The third component uses RCV1-V2, NYT and Web-of-Science, with the training sets subsampled geometrically. Comparing curated topic trees with clustered label trees lets us test the Bertalis et al. reconciliation.

Measurement. For every configuration, we bin leaves by training count and record misrouting rate per bin. From these we estimate $n^*$ and $c$ on the two smallest scales, then predict $\Delta$ at the largest.

Baselines

The naive reference point is a flat classifier over all leaves with the same backbone, data, and tuning budget, which ignores the hierarchy. It also serves as $E_{\text{joint}}$. If it matches or beats every hierarchical variant at full scale, that is itself the answer: scale substitutes for structure in that domain. The second reference is a hard top-down pipeline, which is the object whose penalty we measure.

The treatments are the error-correcting designs:

  • child-first autoregressive decoding as in RADAr;
  • stacked training on held-out parent predictions as in Munoz et al.;
  • soft routing that keeps the top three coarse branches, in the spirit of the beam search in BeamAggR;
  • hierarchy-preserving embedding regularization as in Ahmed et al.

All variants use the same backbone and data, an equal hyperparameter-search budget measured in trials per scale, and an equal number of training steps per scale.

The strongest existing method differs by domain. On BIOSCAN it is the hierarchy-regularized model of Ahmed et al., which reports 88.0% global top-1 accuracy under combined image and DNA noise. The only clean image-only reference in the file is ViT-B/16 at 99.69% order-level and 97.59% family-level micro top-1 accuracy, from the BIOSCAN-1M paper. The file has no reported result on Lacuna for clean species- or BIN-level accuracy. On RCV1-V2, the strongest is RADAr at 87.23 micro-F1, tied with HBGL. On NYT, CascadeXML reports 84.61 R-Precision.

We count a mechanism as beating these baselines only if it improves on them at matched scale by more than the spread across three random seeds. Reproducing these published numbers at full scale is a prerequisite before any curve is interpreted.

The ablation that isolates our key ingredient is the predictor itself. We compare the frequency-binned prediction of $\Delta$ at full scale with a plain power law fitted to aggregate error on the same small subsets. If the aggregate fit predicts just as well, the frequency-bin decomposition adds nothing.

Why not just…?

Why not just train a flat classifier on all the data and drop the hierarchy? This is a serious threat, and the design treats it as a possible outcome rather than something to argue away. The gain reported by Ahmed et al. comes under heavy noise. Banko-style scaling results in Scaling to Very Very Large Corpora for Natural Language Disambiguation suggest that data often beats design. If $\Delta(P)$ decays to zero and the flat model wins at 903k images, we report that, along with the scale at which it happens. What the flat model cannot do alone is say when the hierarchy stops mattering; that is what the predictor provides. On 670k-label problems, a flat output layer is also expensive, per Bertalis et al.

Why not just use a bigger model? If cascades come from limited capacity, widening the backbone should shrink $\Delta$ at fixed $P$. We run a width sweep on the synthetic testbed and two backbone sizes on BIOSCAN to test this. Our hypothesis predicts that capacity changes the constant $c$ but neither the exponent nor $\Delta_\infty$. Two pieces of evidence make capacity a plausible partial explanation: the underfitting at a billion samples reported in Generating QM1B with PySCFIPU, and the agent-scaling result that stronger models did not remove amplification.

Why not just keep several coarse branches open at inference time? Top-three soft routing is cheap and may capture most of the benefit of the heavier treatments. We include it as a treatment. If it closes $\Delta$ to within seed noise at every scale, the practical recommendation becomes "always route softly," and the heavier mechanisms are unnecessary. That would still be a useful, falsifiable outcome.

Why not just plot overall accuracy against data size? Aggregate curves hide the difference between a shrinking tail term and a fixed floor. The two can look alike over a short range of scales. The aggregate-fit ablation tests directly whether separating them improves extrapolation.

Proposed Experiments and Decisive Checks

Four checks are decisive.

  1. Exponent. On the synthetic testbed, the fitted exponent of $\Delta(P)$ should track $a/(1+a)$ across the Zipf sweep, while the ambiguity knob moves only $\Delta_\infty$. A clearly different exponent would falsify the counting argument.
  2. Extrapolation. On BIOSCAN and the text hierarchies, frequency-binned estimates from the 50k and 200k subsets (or their text equivalents) should predict full-scale $\Delta$ more accurately than the aggregate fit. We measure this as absolute error in percentage points against the observed value.
  3. Crossover. For each error-correcting mechanism, we locate the scale at which its advantage over the hard pipeline falls within seed noise, and check whether that scale is ordered by the estimated $n^*$.
  4. Noise floor. Adding the BIOSCAN image and DNA corruptions used by Ahmed et al. should raise $\Delta_\infty$. It should also restore the advantage of the curated taxonomy over a data-clustered tree at full scale, which reconciles their findings with those of Bertalis et al. If the curated tree wins even on clean data at full scale, the reconciliation fails.

As a secondary use of the curves, we apply the conformal method of Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity at each scale. This tests whether prediction-set size in the tail shrinks along the same curve. That paper reports average sets of about 44 at complexity 3 versus about 123 at complexity 1 on PlantCLEF 2015.