Classifiers that know a class taxonomy, such as a tree grouping species into genera, families and orders, can make "better mistakes." When they are wrong, they confuse a husky with a malamute rather than with a toaster. There are two main ways to get this today. One changes the training loss so that coarse errors cost more. This buys milder mistakes but costs accuracy and distorts the model's probabilities. The other leaves training alone and, at test time, picks the class with the lowest expected taxonomic cost under the model's predicted probabilities. This keeps accuracy, but it can only be as good as those probabilities, and they are least reliable for rare classes.
We propose a third route. Keep the training objective exactly as it is: plain conditional likelihood, meaning the log-probability of the correct fine-grained label given the image, which is ordinary cross-entropy. Change only how the output layer is parameterized. Each fine-grained class's weight vector becomes a sum of vectors along its root-to-leaf path, so rare species borrow statistical strength from their genus and family. Then apply the cost-aware test-time rule on top. In two sentences, the hierarchy should shape the structure of the conditional model rather than the objective. Classical results on discriminatively trained graphical models predict that this keeps probabilities honest while making them better where the taxonomy is informative. It should also degrade gracefully where the taxonomy is wrong.
Core Claim
The intersection of these two directions has so far been worked at the level of the loss or the decision rule. Hierarchical cross-entropy (HXE) and soft labels, from Making Better Mistakes, rewrite the conditional likelihood along the taxonomic path with a depth discount. They lose top-1 accuracy: on iNaturalist-H, soft labels with $\beta=10$ raise top-1 error from 43.77% to 55.16% while lowering mistake distance from 2.41 to 2.10. No Cost Likelihood Manipulation at Test Time shows that conditional risk minimization (CRM) applied post hoc to a plain cross-entropy model keeps baseline accuracy. It also proves that CRM never overrides a prediction whose top probability exceeds 0.5. CRM, however, treats the network as frozen, so it inherits whatever miscalibration the network has on tail classes.
Our claim is that the missing piece is a hierarchy-structured conditional model trained by unmodified conditional likelihood. It should reduce the severity of tail-class mistakes under CRM at no cost in top-1 accuracy or calibration. We also claim its behaviour under a wrong taxonomy follows the pattern that Structural Extension to Logistic Regression established for discriminative learning in Bayesian networks. That paper found such learning is robust when the imposed structure is an approximation but overfits when the structure is too complex.
This is a different mechanism from the closest work. HXE changes the objective and CRM changes only the decision. We change neither and instead restrict how parameters are shared.
The Model
Let $f(x)\in\mathbb{R}^d$ be the backbone features of image $x$. Let $\mathcal{A}(y)$ be the set of taxonomy nodes on the path from the root to leaf $y$, and let $v_n\in\mathbb{R}^d$ be a learned vector for node $n$. We set the logit of leaf $y$ as
$$z_y(x) = \Big(\sum_{n\in\mathcal{A}(y)} v_n\Big)^{\top} f(x), \qquad \mathcal{L}(\theta) = -\sum_i \log \frac{e^{z_{y_i}(x_i)}}{\sum_{y'} e^{z_{y'}(x_i)}},$$
and add a weight penalty $\sum_n \lambda_{\text{depth}(n)}\lVert v_n\rVert^2$ with one strength per depth.
With $\lambda=0$ this is just a reparameterized flat softmax. Its content lies in the penalty, which acts as a hierarchical shrinkage prior. A leaf with few examples is pulled toward its ancestors' shared direction rather than toward zero. The model's probability for any ancestor is simply the sum of its leaves' probabilities. The model therefore remains a single normalized distribution over leaves, trained by the same objective whose calibration Karthik et al. rely on.
The per-depth strengths are chosen by validation conditional log-likelihood, not by mistake severity, so the hierarchy cannot "game" the severity metric. This choice responds to the overfitting warning in Greiner et al.: if deep nodes add more structure than the data supports, validation likelihood will push their $\lambda$ up and switch them off.
The same reasoning motivates the design by analogy. Greiner et al. show that conditional-likelihood training of a naive-Bayes-structured network is exactly logistic regression. In that setting structure constrains the conditional model without making it generative. Likewise, Discriminative Learning of Sum-Product Networks and Discriminative Structure Learning of Arithmetic Circuits show that structure paired with a conditional objective yields smaller, more accurate models. Those works treat the label space as flat. We borrow their lesson and apply it to a structured label space.
Measuring Error Propagation Directly
Mistake severity averages away where along the taxonomy a prediction goes wrong. For each error we record its divergence depth, the deepest ancestor shared by the predicted and true leaves. We report the histogram of divergence depths separately for head and tail classes. We also report expected calibration error of the implied ancestor probabilities at every level. The second measure tests whether the miscalibration at intermediate levels attributed to HXE is avoided.
This per-level view makes the cascade question from the hierarchical-error literature measurable in a flat classifier. Stacked and pipelined systems such as Stacked Hierarchical Labeling and Solving the Problem of Cascading Errors fight cascades that come from committing to a coarse decision early. Our model never commits. The question is whether shared parameters move tail-class errors toward deeper divergence, meaning the mistakes happen lower in the tree.
Research Program
We first reproduce flat cross-entropy, HXE and soft labels on tieredImageNet-H and iNaturalist-H, using one backbone and one training schedule across all methods. We then train the path-additive model under the same schedule and apply CRM to every model's probabilities.
To stress the tail, we repeat the main comparison on iNat2017 from The iNaturalist Species Classification and Detection Dataset. Its most-frequent to least-frequent class ratio is 435, versus 1.78 for ImageNet, and the paper reports 67.7% top-1 with Inception ResNet V2 SE. For iNat2017 we build the taxonomy from its species labels.
Finally, we run the misspecification study. We corrupt the taxonomy by reassigning a controlled fraction of leaves to random genera and track how each method degrades.
Baselines
The naive reference point is flat cross-entropy with argmax prediction. Making Better Mistakes reports 27.55% top-1 error on tieredImageNet-H and 43.77% on iNaturalist-H for this baseline, with mistake distance 2.41 on iNaturalist-H. We treat these figures as anchors. The fair comparison is against our own reproduction under the identical schedule.
The strongest existing method is cross-entropy followed by CRM, from No Cost Likelihood Manipulation at Test Time. It is the strongest competitor because it already matches baseline accuracy while improving mistake severity, at zero training cost. The file contains no per-dataset numeric table for CRM; it is reported to keep cross-entropy's top-1 accuracy while lowering mistake severity. We will therefore produce its numbers ourselves from the same cross-entropy checkpoints. Beating it means lower mistake severity and a deeper average divergence depth on tail classes at top-1 error within seed noise of CE+CRM.
We also include a calibrated variant: temperature-scaled cross-entropy followed by CRM, with the temperature fitted on the same validation split we use to tune $\lambda$. This isolates whether simple recalibration closes the gap. The training-time hierarchy losses serve as reference competitors, run with and without CRM. They are HXE with $\alpha=0.1$ (reported 27.68% top-1 error on tieredImageNet-H) and soft labels with $\beta=10$ (reported 55.16% error and 2.10 mistake distance on iNaturalist-H). All methods receive the same hyperparameter search budget on the same validation split.
Two ablations isolate the key ingredient. The first uses path-additive logits over a randomly permuted taxonomy with the same shape. If this matches our results, the gain comes from extra regularization rather than the hierarchy. The second trains path-additive logits with HXE instead of cross-entropy. This tests whether the objective, not the parameterization, is responsible for any calibration difference.
Why not just…?
Why not just temperature-scale the cross-entropy model and run CRM? This is the most serious threat. If the problem with CRM on tail classes were only a global over- or under-confidence, a single temperature would fix it. Temperature, however, rescales all classes together. It cannot tell a rare species that its probability should move toward its genus's typical images. The calibrated-CRM baseline, broken out by head versus tail, settles this. If it matches our method on tail-class severity and per-level calibration, then parameter sharing adds nothing, and we would report that honestly.
Why not just use HXE or soft labels, perhaps with CRM on top? The reported iNaturalist-H trade-off, 11.39 points of added top-1 error for a 0.31 drop in mistake distance, shows these losses buy severity by moving probability mass. Karthik et al. attribute part of the apparent gain to making more but milder mistakes. CRM on top of a miscalibrated model inherits that distortion. We include HXE+CRM and soft labels+CRM as baselines, and the per-level calibration plots will show whether their intermediate-level probabilities are worse than ours.
Why not just a coarse-to-fine cascade with soft routing or sampling? Pipelines of the kind analysed by Finkel et al. gain modestly from integrating over upstream uncertainty: 0.7 to 1.2 F-measure points for semantic role labeling, and textual-entailment accuracy up from 59.13% to 60.88%. They still train stages independently and pass information in one direction only. Our model is a single jointly trained conditional distribution with no routing decision to get wrong, so it avoids the cascade rather than patching it. We do not claim superiority to learned cascades on accuracy grounds without testing. A cascade baseline is secondary here and would be added only if the main comparison is positive.
Decisive Checks
The proposal succeeds if three things happen on iNaturalist-H and iNat2017 tail classes. Path-additive cross-entropy with CRM must lower mistake severity and shift divergence depth deeper relative to calibrated CE+CRM. Top-1 error and ancestor-level calibration error must be no worse. The permuted-taxonomy ablation must lose the gain.
Under taxonomy corruption, Greiner et al.'s account predicts a specific pattern. The learned $\lambda$ at corrupted levels should grow, and performance should decay toward flat cross-entropy rather than below it. HXE, whose objective hard-codes the tree, should fall below flat cross-entropy.
If, instead, gains appear on head classes as much as tail classes, or survive permutation, the mechanism is generic regularization and the hierarchy claim fails. A null result against calibrated CRM would be the most informative outcome of all. It would imply that post-hoc risk minimization plus recalibration is already the right way to combine conditional likelihood with taxonomies.
References
- Making Better Mistakes: Leveraging Class Hierarchies With Deep Networks
- No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks
- Structural Extension to Logistic Regression: Discriminative Parameter Learning of Belief Net Classifiers
- Discriminative Learning of Sum-Product Networks
- Discriminative Structure Learning of Arithmetic Circuits
- Stacked Hierarchical Labeling
- Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines
- The INaturalist Species Classification and Detection Dataset