No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks

Shyamgopal Karthik, Ameya Prabhu, Puneet K. Dokania, Vineet Gandhi

Introduction

In many real-world applications of deep learning, such as autonomous driving or medical diagnostics, not all errors are created equal. Mistaking a pedestrian for a cyclist is a significantly less severe error than mistaking a pedestrian for a background object. However, standard deep learning models typically employ a "flat" classification approach using cross-entropy loss, which treats every incorrect class as equally wrong.

While various "hierarchy-aware" training methods have been proposed to address this by incorporating taxonomic structures (like WordNet), they often suffer from significant drawbacks:

  1. Metric Gaming: Existing metrics for "mistake severity" can be misleading. A model can appear to make "better" mistakes simply by making more low-severity mistakes, which often leads to a significant drop in overall top-1 accuracy.
  2. Calibration Issues: Many specialized hierarchical models are poorly calibrated, meaning their confidence scores do not reflect actual likelihood, making them unreliable for safety-critical tasks.

This paper demonstrates that a simple, decades-old post-hoc technique called Conditional Risk Minimization (CRM) can significantly improve mistake quality at test time with zero retraining and almost no loss in accuracy.

Methodology: Conditional Risk Minimization

The authors propose a "no-cost" solution by applying CRM at inference time. Unlike methods that modify the training objective, CRM works with any standard pre-trained model.

Figure 1: Illustration of the CRM process. A hierarchy defines the cost matrix C. The model's predicted probabilities P are multiplied by C to find the expected risk R for each class.

The process involves three main steps:

  1. Likelihood Estimation: A standard model provides a probability distribution (likelihoods) $P$ over all $K$ classes.
  2. Cost Matrix Construction: A matrix $C$ is defined where $C_{i,j}$ represents the cost of predicting class $i$ when the true label is $j$. The cost is typically the height of the Lowest Common Ancestor (LCA) in the label hierarchy.
  3. Risk Minimization: Instead of selecting the class with the highest probability, CRM selects the class $k$ that minimizes the expected risk: $$R(y=k|x) = \sum_{j=1}^{K} C_{k,j} \cdot p(y=j|x)$$

Essentially, CRM "nudges" the model toward predictions that may have slightly lower probabilities but significantly lower expected costs if incorrect. A key theoretical contribution (Theorem 1) proves a "confidence safeguard": if a model is more than 50% confident in its top prediction, CRM will not change that prediction. This explains why CRM maintains high top-1 accuracy while other methods see significant drops.

Experimental Setup

The approach was evaluated on two large-scale, hierarchy-aware benchmarks:

  • tieredImageNet-H: A version of ImageNet with classes organized by the WordNet ontology.
  • iNaturalist-H: A fine-grained classification dataset of plants and animals based on biological taxonomy.

The authors also critique the standard "Average Mistake Severity" metric, noting it only considers incorrect samples. They advocate for Hierarchical Distance@k, which evaluates the ranking of all top predictions to provide a more holistic view of model performance.

Key Results

CRM consistently outperforms state-of-the-art (SOTA) hierarchy-aware methods like Hierarchical Cross-Entropy (HXE) and Soft-labels across several dimensions.

Figure 6: Comprehensive results on tiered-ImageNet and iNaturalist19. (a) Mistake severity vs. top-1 error; (b) Distribution of mistake distances; (c) Average top-1 hierarchical distance vs. top-1 error. CRM (green X) maintains high accuracy while achieving superior mistake metrics.
  • Superior Ranking: CRM achieves better Average Hierarchical Distance@k, meaning its secondary and tertiary guesses are much closer to the ground truth than those of competing methods.
  • Accuracy Preservation: While SOTA hierarchy-aware models often saw top-1 accuracy drops of 2-5%, CRM typically stayed within 0.5% of the standard cross-entropy baseline.
  • Better Calibration: Because CRM uses the outputs of a standard cross-entropy model (which is generally better calibrated than specialized hierarchical models), it maintains much lower Expected Calibration Error (ECE).

Conclusion and Future Work

CRM provides a simple, effective, and "no-cost" way to improve the safety and reliability of deep networks. Its main limitation is its dependence on a high-quality, pre-defined hierarchy. Future research could explore learning non-uniform edge weights to better reflect real-world costs or integrating CRM-like logic more deeply into the training process without sacrificing calibration or accuracy.

Summary updated

Abstract

There has been increasing interest in building deep hierarchy-aware classifiers that aim to quantify and reduce the severity of mistakes, and not just reduce the number of errors. The idea is to exploit the label hierarchy (e.g., the WordNet ontology) and consider graph distances as a proxy for mistake severity. Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we find that current state-of-the-art hierarchy-aware deep classifiers do not always show practical improvement over the standard cross-entropy baseline in making better mistakes. The reason for the reduction in average mistake-severity can be attributed to the increase in low-severity mistakes, which may also explain the noticeable drop in their accuracy. To this end, we use the classical Conditional Risk Minimization (CRM) framework for hierarchy-aware classification. Given a cost matrix and a reliable estimate of likelihoods (obtained from a trained network), CRM simply amends mistakes at inference time; it needs no extra hyperparameters and requires adding just a few lines of code to the standard cross-entropy baseline. It significantly outperforms the state-of-the-art and consistently obtains large reductions in the average hierarchical distance of top-$k$ predictions across datasets, with very little loss in accuracy. CRM, because of its simplicity, can be used with any off-the-shelf trained model that provides reliable likelihood estimates.
BibTeX
@inproceedings{karthik2021no,
  author = {Shyamgopal Karthik and Ameya Prabhu and Puneet K. Dokania and Vineet Gandhi},
  title = {{No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=193sEnKY1ij},
}
Figure