Train the Level Where Mistakes Start: Origin-Aware Loss Weighting for Hierarchical Classifiers

Many classifiers predict a label at several levels of a taxonomy at once, for example family, genus and species of a butterfly. They are trained with one loss per level, added together with some weights. When the species prediction is wrong, the cause is often that the genus or family prediction was already wrong. The species error was inherited, not created at the species level. Current methods that adjust loss weights during training, such as GradNorm and CoBa, cannot tell these cases apart. They see a high species loss, decide species is lagging, and push more weight onto species. If the mistake actually starts higher in the tree, that extra weight goes to the wrong place.

We propose to weight each level by the errors it originates, not the errors it shows. A wrong leaf prediction has a single "first wrong level" on its path. Counting where those first failures occur splits leaf error exactly into contributions from each depth. A cheap controller that needs only scalar statistics then moves loss weight toward the depth where failures are currently being born. We think this is the missing link between two literatures. Work on hierarchical error propagation has shown that early-level mistakes cascade and cannot be recovered. Work on dynamic task weighting has built controllers that respond to training pace, but it treats tasks as interchangeable.

Core Claim

Flat pace-based weighting misreads inherited error as local lag. A controller driven by an exact decomposition of leaf error into per-level origin terms should do three things. It should improve fine-grained accuracy over flat dynamic weighting and over static hierarchical schedules. It should reduce the share of leaf errors that are inherited. It should also make the classifier more robust when decoded top-down, where cascading errors cannot be undone.

The decomposition is the following. Take a tree of depth $D$ and a held-out set. Let $c_{d}$ be the fraction of samples whose predictions are correct at every level from 1 through $d$, with $c_0 = 1$. The origin error at level $d$ is $o_d = c_{d-1} - c_d$, the fraction of samples whose path first breaks at level $d$. The leaf error then splits exactly as

$$1 - c_D = \sum_{d=1}^{D} o_d .$$

Each $o_d$ is the part of the final error that level $d$ is responsible for. A flat controller tracks instead the per-level error $1 - \text{acc}_d$, which mixes $o_d$ with all errors inherited from levels above. For training we use a smoothed version: hard correctness is replaced by the model's probability on the true label at each level, multiplied along the path. This gives the controller a less noisy signal.

Positioning Against Prior Work

The evidence that motivates this comes from the cascading-error literature. In multi-level retrieval, a root misclassification sends all later search into the wrong sub-space (Brodley et al.). In "scan-then-zoom" part localization, a missed coarse detection leaves the fine networks looking at irrelevant pixels (Zhou and Mok). In multi-scale video generation, coarse layout errors get "baked in" and later stages only sharpen them (Castrejón et al.). These papers diagnose where errors come from, but none of them uses the diagnosis to steer training.

Kantor et al. come closest from the hierarchy side (Asymptotic Cross-Entropy Weighting). They show that coarse heads converge early and starve the leaves of gradient, and they fix this with a static asymptotic weighting curve. We keep their diagnosis but replace the fixed curve with a measured, adaptive signal. That signal can also move weight back up the tree when coarse errors resurface, which their curve cannot do.

Hierarchical Class-Based Curriculum Loss has two parts. It bounds each child's loss by its parent's loss with a max operator, and it admits harder classes through a step-function curriculum. Its schedule is set by fixed thresholds, not by where errors originate. Hierarchical loss for classification uses fixed depth weights that halve at each level.

The pace-balancing methods supply our technique. GradNorm equalizes the relative training rates of tasks through gradient norms. CoBa does something similar using slopes of the validation loss, and it also cuts the weight of tasks that begin to overfit. We reuse CoBa's slope-based controller, because it needs no per-task gradient computations. The only change is the signal it reads: we feed it $o_d$ in place of each level's own loss. Both methods treat tasks as flat, and that is exactly the assumption we drop.

One paper constrains our design. Coherent Hierarchical Multi-Label Classification Networks already removes gradient contradictions between parent and child heads through its max-constraint loss. So we do not propose parent-child gradient surgery. Instead we treat that coherent loss as a base that our weighting sits on top of. A second constraint comes from Wu, Tygert and LeCun, who report that plain flat cross-entropy often does as well on hierarchical metrics as their tree loss. That finding is the main reason this proposal begins with a test that can kill it.

Research Program

The first step is diagnostic. We train standard joint multi-head classifiers with equal weights, with GradNorm, and with CoBa. Throughout training we log the origin profile $(o_1,\dots,o_D)$ and the inherited share of leaf error, $1 - o_D/(1-c_D)$.

The hypothesis predicts two things. First, early in training the inherited share should be large, often above half. Second, flat controllers should raise the leaf weight during exactly those epochs. If the inherited share is small at every point of training, then leaf error really is local, flat controllers are reading the right signal, and the proposal should stop there. We will report that negative result.

The second step builds the controller. Every $k$ steps, a forward pass over a fixed validation slice computes the smoothed $o_d$ for each level. We fit slopes to its recent trajectory. The weights follow CoBa's two scores, with $o_d$ substituted for the loss. Levels whose origin error is large and falling slowly get more weight. Levels whose origin error turns upward on held-out data get less, which acts as level-specific early stopping. The weights are renormalized so that they sum to $D$.

The cost is one extra forward pass on a small slice and no extra backward passes. This matters because per-task gradient computation is the known scaling weakness of GradNorm and PCGrad.

The third step tests the mechanism under controlled conditions. On CIFAR-100, with its 20 superclasses as the coarse level, we inject feature-level or label-level corruption that targets only the coarse decision or only the fine decision. In these two regimes, origin-aware and symptom-aware controllers predict opposite weight changes. That makes it a clean test of whether the decomposition, rather than dynamic weighting in general, is doing the work.

As a stretch goal, we test the bottom-up direction on nested named entity recognition, where the task is to find entities nested inside other entities. Here origin runs the other way: a missed inner entity breaks the outer one (A Neural Layered Model for Nested NER). The same decomposition can be applied by reversing the order of the levels.

Baselines

The naive reference point is equal weights across levels with plain cross-entropy at each level, plus flat cross-entropy on leaves alone. Wu, Tygert and LeCun suggest these may already be hard to beat, so both are required. We also run a grid search over static per-level weights. Its total training compute is matched to that of our method plus all its tuning runs, which answers the "just tune it" objection on equal terms.

The strongest existing competitors come from both sides. On the hierarchy side, we run Kantor et al.'s asymptotic weighting. It reports 81.01% top-1 accuracy on eButterfly (Asymptotic Cross-Entropy Weighting). We reproduce it in our own pipeline and consider the reproduction valid only if it lands within about a point of that figure.

On ETHEC, the strongest number in the file is the hyperbolic-cone model at 0.5977 species-level F1, against a flat baseline of 0.4578 (Hierarchical Image Classification using Entailment Cone Embeddings). We treat 0.5977 as the bar for ETHEC species F1. On iNaturalist Birds, the HCL curriculum loss reports a hierarchical distance of 0.230, against 0.299 for cross-entropy (Hierarchical Class-Based Curriculum Loss), and we treat that as the bar there.

The most direct competitor is CoBa and GradNorm applied per level to the same network. The file contains no reported result for either method on a taxonomic classification benchmark, so we run both ourselves. All controllers share the same backbone, data splits, epochs and validation slice, and each gets an equal hyperparameter-search budget of, for example, eight configurations.

The key ablation is the same controller fed unconditional per-level error ($1-\text{acc}_d$) in place of $o_d$. It is identical in every respect except the decomposition, so it isolates the core ingredient. A second ablation runs our weighting on top of the C-HMCNN coherent loss. That test asks whether origin-aware weighting adds anything once parent-child contradictions have already been removed.

Beating the baselines means three things. Leaf accuracy or F1 must beat the unconditional-error ablation and per-level CoBa, with gains outside the seed-to-seed spread over at least five seeds. Results must match or beat the reported numbers above. The inherited share of errors must be measurably lower.

Why not just…?

Why not just run CoBa or GradNorm on the levels as if they were ordinary tasks? This is the central threat, and it may partly work. Any controller that eventually boosts lagging leaves will help with the gradient starvation Kantor et al. describe. Our claim is narrower. When error is inherited, a flat controller boosts the wrong level, and this should show up as slower coarse convergence early in training and a higher inherited share of leaf errors. The unconditional-error ablation and the CIFAR-100 targeted-corruption regimes test exactly this. If the flat controller matches ours in both, we will report that the decomposition is unnecessary.

Why not just use Kantor et al.'s static asymptotic curve, or HCL's curriculum? A fixed schedule encodes an assumption that coarse levels are learned first and stay learned. That fails when coarse errors return late in training or differ across datasets. Kantor's curve also never measures training pace, which is the gap the file identifies. Our measure is a schedule that adapts per dataset. If the static curves match it on the clean benchmarks, the remaining question is robustness, and the coarse-corruption regime should separate them.

Why not just use flat cross-entropy, given that Wu, Tygert and LeCun found it competitive? Their result is the reason the diagnostic step comes first. If flat training already has a small inherited-error share, our method has nothing to fix. The ETHEC gap between the flat model (0.4578) and the hyperbolic model (0.5977) suggests flat training leaves room for improvement on long-tailed taxonomies, but we treat that as a hypothesis, not a premise.

Why not just use the C-HMCNN coherent loss and stop? Coherence and weighting address different problems. The coherent loss forces parent and child predictions to be logically consistent, but it says nothing about which depth gets training effort. The stacked ablation measures whether the two combine. If coherence alone captures the gain, that is a clean negative finding about weighting.

Proposed Experiments

The main benchmarks are ETHEC (4 levels, about 48k images), eButterfly (3 levels, 100 species), iNaturalist Birds, and CIFAR-100 with its superclasses for the controlled studies. All use a shared convolutional backbone with one head per level.

The metrics are leaf top-1 accuracy and F1, per-level F1, the hierarchical distance of mistakes, the full origin profile with the inherited share, and coherence rate. We also report leaf accuracy under top-down decoding, where each level's prediction restricts the next. That measures how much each training method leaves the cascade exposed to early errors.

Three outcomes decide the proposal. The first is the diagnostic inherited-share curve, which can stop the work at the start. The second is the gap between our controller and its unconditional-error ablation, which shows whether the decomposition matters. The third is the comparison under targeted corruption: if our controller moves weight toward the corrupted level and the flat controller moves it away, the mechanism is confirmed even where headline gains on clean data are small. Wall-clock overhead is reported against GradNorm, whose own overhead the file puts at about 5% on NYUv2.