Standard deep learning models suffer from a fundamental "closed-world" bias. When a classifier trained to recognize dogs and cats is shown a picture of a crab, it doesn't say "I don't know." Instead, as Haoqi Wang and Zhizhong Li observe in ViM: Out-Of-Distribution with Virtual-logit Matching, it will happily classify that crab as a "clapping hand" with high confidence.
This phenomenon, known as the overconfidence gap, occurs because discriminative models are designed to partition a feature space into known categories. Every point in that space—even points far away from any training data—must belong to one of those partitions. This creates "unknown unknowns": regions where the model is wildly confident but entirely wrong.
The Softmax Illusion
The root of the problem lies in the softmax function. Softmax is a relative operator; it forces the model's internal scores (logits) to sum to one. If a model sees an input that looks nothing like its training data, but produces a slightly higher logit for "Class A" than "Class B," the softmax function will amplify that tiny difference into a high-probability prediction.
To combat this, researchers like Derek Hoiem and Zhizhong Li 0001 have explored G-Distillation in G-Distillation: Reducing Overconfident Errors on Novel Samples. The intuition here is that while a single model is prone to overconfidence, an ensemble of models will likely disagree on unfamiliar data. By distilling the diverse uncertainty estimates of an expensive ensemble into a single efficient model using unsupervised data, we can teach that model to "know what it doesn't know."
Pruning the Noise from Unfamiliar Data
One reason models are overconfident on out-of-distribution (OOD) data is that they rely on "noisy" weights—parameters that were useful for fine-tuning on the training set but provide misleading signals for novel inputs.
In DICE: Leveraging Sparsification for Out-of-Distribution Detection, Yiyou Sun and Yixuan Li introduce Directed Sparsification. They found that by selectively pruning the least "contributing" weights in the final layer, they could significantly reduce the noise generated by OOD inputs. The intuition is that in-distribution (ID) data relies on a core set of robust features, while OOD data triggers random, low-magnitude weights that sum up to high-confidence errors. Removing these "weak links" clarifies the boundary between known and unknown.
The Geometry of Knowns
Recent work has shifted from looking at model outputs (logits) to looking at the geometry of the feature space itself. Mouïn Ben Ammar and colleagues leverage a phenomenon called Neural Collapse in NECO: NEural Collapse Based Out-of-distribution detection.
During the final stages of training, in-distribution data points tend to cluster into tight "rays" or subspaces that are maximally distant from one another. NECO works because OOD data naturally falls outside these rigid geometric structures. By measuring how orthogonal a new sample is to these established clusters, the model can identify novelty without ever having seen an outlier during training. Similarly, the PRISM framework in Pseudo-label Induced Subspace Representation Learning for Robust Out-of-Distribution Detection forces ID data into low-dimensional subspaces, making anything that deviates from that manifold immediately suspicious.
Modeling Density and Energy
Another powerful approach is to move from purely discriminative models to hybrid models that understand the density of the input data. If a model knows $p(x)$—the probability of seeing a particular input—it can flag low-density regions as OOD.
- Energy-Based Models: Hu Ding and Qi Chen proposed DEBO in Dual Energy-Based Model with Open-World Uncertainty Estimation for Out-of-distribution detection. By using an energy function to map the training distribution, the model assigns low "energy" to familiar data and high energy to anomalies, providing a mathematically grounded way to signal uncertainty.
- Invertible Features: In Hybrid Models with Deep and Invertible Features, Eric Nalisnick uses normalizing flows to create invertible feature extractors. This allows the model to compute the exact likelihood of an input $p(x)$ alongside the classification $p(y|x)$, effectively checking if the input "belongs" before attempting to label it.
Context and Class Priors
Finally, robustness often requires looking at the broader context of the data. Xue Jiang and Feng Liu point out in Detecting Out-of-distribution Data through In-distribution Class Prior that OOD data isn't just "random"; it often gravitates toward the most common classes in the training set (the "head" classes). By adjusting for these class priors, we can prevent the model from defaulting to its most frequent labels when it encounters something new.
For real-world deployment, researchers like Matteo Guarrera suggest Class-wise Thresholding. Instead of one global "uncertainty" score, the system maintains a specific threshold for every category, acknowledging that some classes (like "dog") are naturally more diverse and prone to overlap with the "unknown" than others.