Introduction
Modern deep learning models suffer from a persistent "confidently wrong" problem. Despite their high accuracy on standard benchmarks, neural networks are notoriously overconfident, often assigning high probabilities to incorrect predictions, out-of-distribution (OOD) data, or nonsensical noise. This paper introduces Introspection-Net, a secondary network designed to monitor the primary model's internal state to accurately predict classification correctness.
The Problem: The Softmax Information Bottleneck
The standard softmax layer, which converts raw network outputs into a probability distribution, acts as an information bottleneck. By normalizing outputs, it exaggerates small relative differences between classes while discarding the absolute magnitude of the network's internal activations. This leads to high-confidence errors in three main scenarios:
- Regular Errors: Misclassifications of standard, in-distribution data.
- Out-of-Distribution (OOD) Data: Inputs the model has never seen (e.g., random noise or images from a different domain).
- Adversarial Examples: Images engineered with subtle noise specifically to trick a model into a high-confidence wrong answer.
Methodology: Introspection-Net
The authors propose a technique called introspection. Instead of relying on the final probability score, they train a small auxiliary model called Introspection-Net to look at the raw "logits" (the values produced by the last layer before the softmax function) of a pre-trained, frozen classifier.
Architecture and Training
The Introspection-Net is a simple 3-layer dense neural network (128 neurons per layer) utilizing ReLU activations, Dropout, and Batch Normalization. Its task is to perform a regression that predicts a "confidence value" between 0 (incorrect) and 1 (correct). To ensure robustness, the network is trained on a balanced dataset containing correctly classified images and various "misclassifications" generated via data augmentation (black patches), OOD datasets, and adversarial attack methods like the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM).
Why Logit Magnitude Matters
The core insight of the paper is that correctly classified images consistently produce higher-magnitude activation patterns than errors or adversarial examples. While adversarial attacks are highly effective at "fooling" the final softmax output by manipulating relative differences, they often fail to mimic the high-magnitude logit patterns characteristic of genuine, clean data.
As shown in the scatter plots above, softmax values for clean data (MNIST) and errors are often clustered near 1.0, making them indistinguishable. However, looking at the raw logit values reveals a distinct separation, allowing the Introspection-Net to catch discrepancies that the primary model misses.
Experimental Results
The researchers evaluated their approach using two primary setups: a custom CNN for MNIST and a Wide Residual Network (WR-28-8) for CIFAR-10. Introspection-Net was compared against the standard "Softmax Baseline" and the "Trust Score" (a popular 1-NN based metric).
Key Findings
- OOD Detection: In MNIST experiments, Introspection-Net achieved a 0.0% False Positive Rate (FPR) at a 95% True Positive Rate for Gaussian noise. In contrast, the Softmax Baseline had a 100% FPR, meaning it could not distinguish noise from digits at all.
- Adversarial Robustness: For the BIM attack on CIFAR-10, Introspection-Net reached an AUROC of 97.9%, compared to a dismal 14.4% for the Softmax Baseline.
- Generalization: The method proved effective across different architectures, suggesting that logit distribution patterns are a fundamental property of how these networks learn.
Limitations and Future Work
Despite its success, Introspection-Net has limitations. It currently requires adversarial training on the specific types of errors it is meant to detect; if a new, unknown attack is used, the network might fail to recognize the pattern. Additionally, unlike model-agnostic methods like Trust Score, Introspection-Net is tied to the specific logits of a single pre-trained model. Future research will explore whether introspection can be achieved using only in-distribution data and how these principles apply to other domains like Natural Language Processing.
Summary updated