Abstract
Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes. To make future work in real-world settings possible, we create new benchmarks for three large-scale settings. To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.
Introduction
Machine learning models often exhibit overconfidence when encountering out-of-distribution (OOD) data—inputs that differ significantly from the training distribution. In safety-critical domains like autonomous driving, this overconfidence can lead to catastrophic failures. Despite the importance of OOD detection, research has historically been confined to small-scale, low-resolution datasets like CIFAR-10. These "toy" problems fail to capture real-world complexities such as fine-grained visual similarities, high-resolution scenes, and the need for localized anomaly detection within multi-label or segmentation tasks.
This work addresses these gaps by scaling OOD detection to massive multi-class, multi-label, and image segmentation tasks, introducing new benchmarks and a robust baseline method.
Methodology: The Power of MaxLogit
The standard approach for OOD detection utilizes Maximum Softmax Probability (MSP). MSP normalizes raw model outputs (logits) into a probability distribution. However, in large-scale settings with thousands of classes, probability mass is often dispersed among many visually similar categories, squashing the MSP score even for familiar objects.
We propose MaxLogit as a superior alternative. MaxLogit simply uses the raw, unnormalized score of the most likely class. Because logits are not normalized by the total number of classes, they provide a cleaner signal of model familiarity, especially in high-dimensional label spaces.
New Benchmarks for Realistic OOD Detection
To facilitate research in realistic settings, we introduce several high-quality benchmarks:
- The Species Dataset: A massive multi-class OOD test set with over 700,000 images across 1,300+ anomalous biological species. It is designed to be disjoint from ImageNet-21K, ensuring rigorous testing without data leakage.
- CAOS (Combined Anomalous Object Segmentation): This benchmark focuses on localized anomalies within a scene. It includes StreetHazards, a simulated driving dataset where diverse 3D models (e.g., helicopters) are integrated into road scenes with realistic lighting, and BDD-Anomaly, a real-world dataset derived from BDD100K.
Experimental Results
Our experiments demonstrate that MaxLogit consistently outperforms more complex state-of-the-art methods across all tasks:
- Multi-class: On ImageNet-21K, MaxLogit achieved a mean AUROC of 63.2%, compared to 54.6% for MSP.
- Multi-label: On COCO, MaxLogit reached an AUROC of 90.3%, significantly higher than the 70.7% achieved by MSP.
- Segmentation: On StreetHazards, MaxLogit reduced the False Positive Rate (FPR95) to 26.5%, outperforming MSP (33.7%) and various specialized architectures.
Conclusion
Scaling OOD detection reveals that simple baselines like MaxLogit can be remarkably effective. While challenges remain—particularly regarding false positives at semantic object boundaries—this work provides the benchmarks and baselines necessary to move OOD detection research toward real-world applicability. Future work should focus on refining localized detection and handling fine-grained anomalies.
Summary updated