Abstract
To classify in-distribution samples, deep neural networks learn label-discriminative representations, which, however, are not necessarily distribution-discriminative according to the information bottleneck. Therefore, trained networks could assign unexpected high-confidence predictions to out-of-distribution samples drawn from distributions differing from that of in-distribution samples. Specifically, networks extract the strongly label-related information from in-distribution samples to learn the label-discriminative representations but discard the weakly label-related information. Accordingly, networks treat out-of-distribution samples with minimum label-sensitive information as in-distribution samples. According to the different informativeness properties of in- and out-of-distribution samples, a Dual Representation Learning (DRL) method learns distribution-discriminative representations that are weakly related to the labeling of in-distribution samples and combines label- and distribution-discriminative representations to detect out-of-distribution samples. For a label-discriminative representation, DRL constructs the complementary distribution-discriminative representation by an implicit constraint, i.e., integrating diverse intermediate representations where an intermediate representation less similar to the label-discriminative representation owns a higher weight. Experiments show that DRL outperforms the state-of-the-art methods for out-of-distribution detection.
Introduction
Deep neural networks (DNNs) are often "confidently wrong" when they encounter data they weren't trained on, a phenomenon known as Out-of-Distribution (OOD) detection failure. Standard image classifiers are built to be efficient according to the Information Bottleneck principle, which suggests a model learns to keep only the features strictly necessary for predicting a label (e.g., the shape of a dog's ears) while discarding "irrelevant" information (e.g., background texture).
While this compression aids classification, it hinders OOD detection. As illustrated in Figure 1, a model might focus on label-discriminative features to predict "Dog" with high confidence, even if the sample is a drawing on a bag (OOD). The model lacks the context—the "weakly label-related information"—to realize that the overall distribution of the image is fundamentally different from its training set.
Dual Representation Learning (DRL)
The proposed Dual Representation Learning (DRL) framework introduces an auxiliary network to capture the "discarded" information. By forcing the model to look at the "boring" background details it usually ignores, DRL better distinguishes between familiar and unfamiliar data.
As shown in Figure 4, the DRL framework consists of three main components:
- Pretrained Network ($g_\phi$): A standard classifier that focuses on label-discriminative representations ($d$).
- Auxiliary Network ($f_\theta$): A novel addition trained to capture distribution-discriminative representations ($c$).
- Implicit Constraint: To ensure the auxiliary network doesn't simply replicate the main network, DRL decomposes image features into multiple components and assigns higher weights to those least similar to the label-discriminative features. This forces the auxiliary network to "mine" the information the main network discarded.
During inference, the model calculates an OOD Score ($S$) by evaluating both representations. For an In-Distribution (ID) sample, both representations should point toward the same label. For an OOD sample, the "strong" features might suggest a label, but the "weak" features likely won't, leading to a lower overall confidence score and successful detection.
Experimental Results
The researchers evaluated DRL using standard benchmarks, including CIFAR-10 and Mini-ImageNet as ID datasets, against a wide battery of OOD datasets such as CIFAR-100, CUB200, and COCO.
DRL outperformed state-of-the-art post-hoc methods (like ODIN and Mahalanobis distance) and confidence enhancement methods (like CSI). On CIFAR-10, DRL achieved an average AUROC of 81.9%, compared to roughly 79% for most competing methods. Notably, DRL showed a significant 2.94% improvement in distinguishing "Near OOD" samples, which are notoriously difficult because the classes are semantically similar.
Conclusion
DRL demonstrates that the information a model usually throws away as "noise" is actually the most valuable signal for determining if a sample belongs to the training distribution. By leveraging dual representations, DRL makes model confidence scores more honest and improves safety in real-world deployments. Future work will address the computational overhead of the auxiliary network and explore joint training architectures.
Summary updated