Reducing Overconfident Errors on Novel Samples

This direction is merged into another direction: out-of-distribution overconfidence (model overconfidence). Its last version stays readable and citable here.

Version 1 of 157 papers58 conceptsfrozen 2026-10-04

Standard deep learning models suffer from a fundamental "closed-world" bias. When a classifier trained to recognize dogs and cats is shown a picture of a crab, it doesn't say "I don't know." Instead, as Haoqi Wang and Zhizhong Li observe in ViM: Out-Of-Distribution with Virtual-logit Matching, it will happily classify that crab as a "clapping hand" with high confidence.

This phenomenon, known as the overconfidence gap, occurs because discriminative models are designed to partition a feature space into known categories. Every point in that space—even points far away from any training data—must belong to one of those partitions. This creates "unknown unknowns": regions where the model is wildly confident but entirely wrong.

The Softmax Illusion

The root of the problem lies in the softmax function. Softmax is a relative operator; it forces the model's internal scores (logits) to sum to one. If a model sees an input that looks nothing like its training data, but produces a slightly higher logit for "Class A" than "Class B," the softmax function will amplify that tiny difference into a high-probability prediction.

To combat this, researchers like Derek Hoiem and Zhizhong Li 0001 have explored G-Distillation in G-Distillation: Reducing Overconfident Errors on Novel Samples. The intuition here is that while a single model is prone to overconfidence, an ensemble of models will likely disagree on unfamiliar data. By distilling the diverse uncertainty estimates of an expensive ensemble into a single efficient model using unsupervised data, we can teach that model to "know what it doesn't know."

Pruning the Noise from Unfamiliar Data

One reason models are overconfident on out-of-distribution (OOD) data is that they rely on "noisy" weights—parameters that were useful for fine-tuning on the training set but provide misleading signals for novel inputs.

In DICE: Leveraging Sparsification for Out-of-Distribution Detection, Yiyou Sun and Yixuan Li introduce Directed Sparsification. They found that by selectively pruning the least "contributing" weights in the final layer, they could significantly reduce the noise generated by OOD inputs. The intuition is that in-distribution (ID) data relies on a core set of robust features, while OOD data triggers random, low-magnitude weights that sum up to high-confidence errors. Removing these "weak links" clarifies the boundary between known and unknown.

The Geometry of Knowns

Recent work has shifted from looking at model outputs (logits) to looking at the geometry of the feature space itself. Mouïn Ben Ammar and colleagues leverage a phenomenon called Neural Collapse in NECO: NEural Collapse Based Out-of-distribution detection.

During the final stages of training, in-distribution data points tend to cluster into tight "rays" or subspaces that are maximally distant from one another. NECO works because OOD data naturally falls outside these rigid geometric structures. By measuring how orthogonal a new sample is to these established clusters, the model can identify novelty without ever having seen an outlier during training. Similarly, the PRISM framework in Pseudo-label Induced Subspace Representation Learning for Robust Out-of-Distribution Detection forces ID data into low-dimensional subspaces, making anything that deviates from that manifold immediately suspicious.

Modeling Density and Energy

Another powerful approach is to move from purely discriminative models to hybrid models that understand the density of the input data. If a model knows $p(x)$—the probability of seeing a particular input—it can flag low-density regions as OOD.

Context and Class Priors

Finally, robustness often requires looking at the broader context of the data. Xue Jiang and Feng Liu point out in Detecting Out-of-distribution Data through In-distribution Class Prior that OOD data isn't just "random"; it often gravitates toward the most common classes in the training set (the "head" classes). By adjusting for these class priors, we can prevent the model from defaulting to its most frequent labels when it encounters something new.

For real-world deployment, researchers like Matteo Guarrera suggest Class-wise Thresholding. Instead of one global "uncertainty" score, the system maintains a specific threshold for every category, acknowledging that some classes (like "dog") are naturally more diverse and prone to overlap with the "unknown" than others.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reducing_overconfident_errors_on_novel_s_e308ac19,
  title = {Reducing Overconfident Errors on Novel Samples},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reducing-overconfident-errors-on-novel-samples/rdr_e308ac19135d4b55bac93c640b9a3cae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reducing_overconfident_errors_on_novel_s_e308ac19_v1,
  title = {Reducing Overconfident Errors on Novel Samples},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reducing-overconfident-errors-on-novel-samples/rdr_e308ac19135d4b55bac93c640b9a3cae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}