Abstract
Out-of-distribution (OOD) detection is a critical task for deploying machine learning models in the open world. Distance-based methods have demonstrated promise, where testing samples are detected as OOD if they are relatively far away from in-distribution (ID) data. However, prior methods impose a strong distributional assumption of the underlying feature space, which may not always hold. In this paper, we explore the efficacy of non-parametric nearest-neighbor distance for OOD detection, which has been largely overlooked in the literature. Unlike prior works, our method does not impose any distributional assumption, hence providing stronger flexibility and generality. We demonstrate the effectiveness of nearest-neighbor-based OOD detection on several benchmarks and establish superior performance. Under the same model trained on ImageNet-1k, our method substantially reduces the false positive rate (FPR@TPR95) by 24.77% compared to a strong baseline SSD+, which uses a parametric approach Mahalanobis distance in detection.
Introduction
As machine learning models transition from controlled laboratory environments to "open world" deployments, the ability to identify Out-of-Distribution (OOD) samples becomes paramount. A robust system must distinguish between known categories and novel inputs to avoid making confident but erroneous predictions.
Recent literature has focused on distance-based OOD detection, which utilizes the feature embeddings of a neural network. The core intuition is that OOD samples should reside in low-density regions of the feature space, far from the clusters of in-distribution (ID) training data. However, popular approaches like the Mahalanobis distance rely on the assumption that features follow a specific distribution, such as a mixture of Gaussians. This paper argues that such parametric assumptions are often too restrictive for the complex feature spaces learned by deep neural networks. Instead, we propose KNN-OOD, a non-parametric framework that leverages $k$-nearest neighbor distances to achieve state-of-the-art OOD detection without distributional constraints.
The KNN-OOD Framework
The proposed method is conceptually simple and can be integrated with any pre-trained deep learning model. The detection process follows four primary steps:
- Feature Extraction: For a given input $\mathbf{x}$, we extract the penultimate layer embedding $\phi(\mathbf{x})$.
- Feature Normalization: We project the embeddings onto a unit hypersphere: $\mathbf{z} = \phi(\mathbf{x})/\|\phi(\mathbf{x})\|_2$. This step is critical; without it, the Euclidean distance is often dominated by the $L_2$ norm of the features rather than their semantic direction. We observe that ID data typically exhibits larger feature norms than OOD data, which can confound distance-based metrics if not normalized.
- Nearest Neighbor Search: We compute the Euclidean distance between the test embedding $\mathbf{z}$ and all embeddings in the training set $\{\mathbf{z}_i\}_{i=1}^n$.
- OOD Scoring: The distance to the $k$-th nearest neighbor serves as the OOD score. Samples with a distance exceeding a threshold (calibrated using ID data) are flagged as OOD.
Synergy with Contrastive Learning
While KNN-OOD works effectively with standard cross-entropy trained models, its performance is significantly enhanced when paired with Supervised Contrastive Learning (SupCon). SupCon explicitly encourages the model to learn a feature space where same-class samples are tightly clustered and different-class samples are maximally separated. This creates a cleaner, more discriminative "map" for the KNN search, making OOD samples significantly easier to isolate.
Experimental Results
We evaluated KNN-OOD across diverse benchmarks, ranging from CIFAR-10 and CIFAR-100 to large-scale ImageNet-1k datasets. The OOD test sets included SVHN, LSUN, iSUN, Textures, and Places365, along with "hard" OOD tasks like ImageNet-R.
Performance on ImageNet-1k
On the challenging ImageNet-1k benchmark, KNN-OOD (using a SupCon-trained ResNet-50) demonstrated remarkable gains. Compared to SSD+, a strong baseline that utilizes Mahalanobis distance, our method reduced the False Positive Rate (FPR@TPR95) by 24.77%. This highlights the advantage of a non-parametric approach in high-dimensional, multi-class feature spaces where Gaussian assumptions frequently fail.
Performance on CIFAR
On CIFAR-10, KNN-OOD achieved an average FPR95 of 11.07%, outperforming previous state-of-the-art methods like CSI. The method proved robust to the choice of $k$; on ImageNet, performance remained consistently high for $k$ values ranging from 100 to 1000.
Scalability and Efficiency
A common critique of non-parametric methods is their computational and storage overhead. We address these concerns through two key findings:
- Search Efficiency: By utilizing the Faiss library for similarity search, KNN-OOD can scale to millions of reference images. On ImageNet, the inference time is approximately 10ms per image, making it viable for real-time applications.
- Data Efficiency: We found that storing the entire training set is not strictly necessary. Using only 1% of the training embeddings as a reference set maintains high detection performance while significantly reducing the storage footprint.
Conclusion
KNN-OOD demonstrates that a simple, non-parametric approach to OOD detection can outperform complex parametric alternatives. By removing distributional assumptions and emphasizing the importance of feature normalization, we provide a flexible and powerful tool for safe model deployment. Future research may focus on further compressing the reference database and exploring the theoretical boundaries of non-parametric OOD detection in even more complex, open-world scenarios.
Summary updated