Abstract
Current progress in out-of-distribution (OOD) detection is limited by the lack of large, high-quality datasets with clearly defined OOD categories across varying difficulty levels (near- to far-OOD) that support both fine- and coarse-grained computer vision tasks. To address this limitation, we introduce ICONIC-444 (Image Classification and OOD Detection with Numerous Intricate Complexities), a specialized large-scale industrial image dataset containing over 3.1 million RGB images spanning 444 classes tailored for OOD detection research. Captured with a prototype industrial sorting machine, ICONIC-444 closely mimics real-world tasks. It complements existing datasets by offering structured, diverse data suited for rigorous OOD evaluation across a spectrum of task complexities. We define four reference tasks within ICONIC-444 to benchmark and advance OOD detection research and provide baseline results for 22 state-of-the-art post-hoc OOD detection methods.
Introduction
In real-world applications, machine learning models frequently encounter unexpected inputs that differ from their training data—known as Out-of-Distribution (OOD) samples. A significant vulnerability of modern neural networks is their tendency to classify these unknown samples into known categories with high confidence, potentially leading to catastrophic failures in safety-critical systems.
Building robust OOD detectors requires rigorous benchmarks. However, existing datasets like ImageNet or CIFAR were primarily designed for classification, not OOD detection. These datasets often suffer from "contamination" (where OOD test sets contain In-Distribution objects), lack smooth transitions between "near" and "far" OOD samples, and fail to replicate the controlled yet highly complex environments of industrial applications. ICONIC-444 addresses these gaps by providing a purpose-built, massive dataset designed to stress-test OOD detection methods under realistic conditions.
The ICONIC-444 Dataset
ICONIC-444 is a massive, contamination-free dataset comprising over 3.1 million high-resolution images across 444 classes. The data was captured using a prototype free-fall industrial sorting machine equipped with a high-resolution line-scan camera and controlled lighting, capturing objects like grains, nuts, and foreign materials in mid-air.
The dataset is structured into "Food" (In-Distribution targets) and "Non-Food" (Far-OOD targets such as glass, stones, and plastic). To further challenge models, the authors included:
- Extreme OOD Sets: Samples from standard public datasets like ImageNet, iNaturalist, and Places365.
- Synthetic OOD: 25 types of synthetic corruptions (noise, color shifts) to simulate sensor failures.
As shown in the table below, ICONIC-444 significantly exceeds the scale of previous specialized datasets while being uniquely designed for both Image Classification (IC) and OOD Detection (OODD).
Benchmarking Methodology
The authors defined four classification tasks of increasing complexity: Almond (7 classes), Wheat (12 fine-grained classes), Kernels (29 classes), and Food-grade (324 classes). They benchmarked 22 state-of-the-art post-hoc OOD detection methods using ResNet18 and Compact Transformer (CCT) backbones.
Crucially, the study advocates for stricter evaluation metrics. While standard OOD evaluation uses FPR95 (False Positive Rate at 95% True Positive Rate), the authors argue that a 5% loss of valid data is unacceptable in industrial settings. They propose FPR99 as a more realistic and demanding operating point for safety-critical deployment.
Key Results and Insights
The benchmarking revealed several critical insights:
- Task-Dependent Performance: Unlike results on ImageNet where "model enhancement" methods (e.g., ReAct, DICE) often lead, "feature-based" methods like GRAM, KNN, and ViM performed significantly better on ICONIC-444. This is attributed to the object-centric nature of the dataset, which creates tighter feature clusters compared to the scene-centric variance of ImageNet.
- The FPR99 Reality Check: While AUROC scores appeared promising, the FPR99 metric revealed that most methods allow a massive amount of OOD data to pass through when forced to retain 99% of In-Distribution data. Even on "easy" synthetic OOD samples, many methods exceeded a 30% FPR.
- Top Performers and Architecture Bias: GRAM (utilizing Gram matrices of feature activations) emerged as the top overall performer. Interestingly, ResNet18 generally outperformed the Compact Transformer, likely because many post-hoc methods were originally optimized for CNN feature spaces.
- Scaling Limitations: Increasing model complexity (e.g., using ConvNeXt or CLIP) did not inherently solve OOD detection. CLIP, in particular, struggled because industrial sorting images are fundamentally different from its web-scraped training distribution.
Conclusion
ICONIC-444 provides a rigorous new standard for OOD detection research, highlighting that the field is far from solved for real-world industrial applications. The authors emphasize the need for methods that can handle "Near-OOD" samples—such as confusing rye flakes with peanuts—and urge the community to adopt stricter metrics like FPR99 to ensure research aligns with practical safety requirements. Future work may leverage the dataset's 53 classes of manually induced defects for large-scale anomaly detection research.
Summary updated