TL;DR
Most out-of-distribution (OOD) detection research is stuck on small-scale, low-resolution datasets that don't reflect real-world complexity. This paper scales OOD detection to thousands of classes and high-resolution images, introducing massive new benchmarks and proving that a surprisingly simple method—MaxLogit—outperforms complex state-of-the-art techniques across multi-class, multi-label, and segmentation tasks.
What Problem Does This Paper Address?
Machine learning models are notoriously overconfident. When shown something they’ve never seen before (an OOD example), they often assign it a high-confidence label instead of admitting they don't know. This is a massive safety risk for applications like self-driving cars or medical diagnosis.
The authors argue that the research community has been "playing in a sandbox." Most OOD benchmarks use tiny datasets like CIFAR-10, which have only a few, very distinct classes. In the real world, models deal with thousands of fine-grained categories (e.g., hundreds of different bird species). The authors demonstrate that the standard baseline for OOD detection—Maximum Softmax Probability (MSP)—fails at scale because the probability mass gets "smearing" across many visually similar in-distribution classes, making a perfectly normal image look like an anomaly.
Methods
The paper’s primary technical contribution is the MaxLogit detector.
Standard classifiers use a "softmax" layer to turn raw model outputs (logits) into probabilities that sum to one. MSP uses the highest of these probabilities as a confidence score. However, as the number of classes increases, these probabilities naturally shrink.
MaxLogit ignores the softmax normalization entirely. It simply uses the maximum unnormalized logit value ($max_k f(x)_k$) as the confidence score. Because logits aren't forced to sum to one, they aren't suppressed by the presence of many similar classes. The authors also extend this to:
- Multi-label settings: Where an image can contain multiple objects (e.g., a "bat," "mitt," and "person").
- Anomaly Segmentation: Identifying specific pixels in an image that don't belong (e.g., a lost shipping crate in the middle of a road).
Datasets
To facilitate this large-scale research, the authors introduced several massive new benchmarks:
- Species: A multi-class OOD dataset with over 700,000 images and 1,316 anomalous species. It is specifically designed to be disjoint from ImageNet-21K to ensure rigorous testing without data leakage.
- CAOS (Combined Anomalous Object Segmentation): A benchmark for pixel-level anomaly detection. It combines StreetHazards, a simulated driving dataset where 250 unique 3D models are integrated into scenes, and BDD-Anomaly, a real-world dataset derived from BDD100K.
- Multi-label Benchmarks: Evaluations using ImageNet-21K as OOD data for models trained on PASCAL VOC and MS-COCO.
Key Insights and Ideas
- The "Smearing" Effect: In large-scale settings, a model might be 99% sure an image is a "dog" but split that confidence across 10 different breeds. MSP would see a low 10% confidence for any single breed and flag it as an anomaly. MaxLogit maintains a high score because the raw logit for "dog-ness" remains high.
- ViTs aren't a Silver Bullet: Contrary to recent hype, the authors find that simply scaling up Vision Transformers (ViTs) does not solve OOD detection; they still struggle significantly on the new Species benchmark.
- Simulation Matters for Segmentation: Traditional "cut-and-paste" anomaly benchmarks are too easy because models can detect the sharp edges or lighting mismatches of the pasted object. By using the Unreal Engine for StreetHazards, the authors created anomalies that are "naturally" integrated, making the task much harder and more realistic.
Key Results
MaxLogit consistently established a new state-of-the-art baseline:
- Multi-class: On ImageNet-21K, MaxLogit achieved an AUROC of 63.2%, significantly better than MSP’s 54.6%.
- Multi-label: On PASCAL VOC, MaxLogit reached 90.9% AUROC, dwarfing MSP’s 74.2%.
- Segmentation: On the CAOS benchmark, MaxLogit outperformed complex autoencoder-based methods and prior "confidence branch" techniques, particularly in reducing false positives at semantic boundaries.
Limitations and Future Work
Despite the improvements, the authors note that AUPR (Area Under the Precision-Recall curve) remains low for anomaly segmentation. This is due to the extreme class imbalance—anomalies might only take up 0.1% of the pixels in a driving scene.
Additionally, even MaxLogit struggles with "semantic boundaries"—the edges where one object ends and another begins. The authors suggest that future work should focus on better distinguishing between these "hand-off" regions and actual foreign objects.
Summary updated