Enhancing the Power of OOD Detection via Sample-Aware Model Selection

Feng Xue, Zi He, Yuan Zhang, Chuanlong Xie, Zhenguo Li, Falong Tan

Abstract

Deep neural networks often exhibit overconfidence when presented with out-of-distribution (OOD) data, posing significant safety risks in real-world deployments. While various post-hoc detection methods exist, their performance is highly sensitive to model architecture. This paper proposes ZODE (Zoo-based OOD Detection Enhancement), a framework that utilizes a "model zoo" to improve detection robustness. By applying the Benjamini-Hochberg statistical procedure to a diverse set of models, ZODE performs sample-aware model selection. Our approach significantly reduces false positive rates on standard benchmarks, demonstrating the power of collective intelligence in OOD detection.

Introduction

A major challenge in deploying deep learning systems in environments like autonomous driving or medical diagnostics is the "Out-of-Distribution" (OOD) detection problem. Models must identify when an input differs significantly from their training data to avoid making dangerous, overconfident predictions.

Existing state-of-the-art methods typically rely on a single pre-trained architecture. However, research indicates a "model selection dilemma": a model optimized for one type of OOD data (e.g., blurry textures) may fail on another (e.g., novel objects). The performance gap between the best and worst models on certain benchmarks can reach over 32%. Since the nature of OOD data encountered in the wild is unpredictable, picking a single model beforehand is inherently risky.

The ZODE Framework

To address this, we introduce ZODE, which moves away from the single-model bottleneck toward a "collective intelligence" approach. The framework consists of three main components:

  1. The Model Zoo: ZODE aggregates a diverse collection of models, including ResNets, DenseNets, and Swin Transformers, trained using various strategies (e.g., supervised and contrastive learning).
  2. P-Value Normalization: To compare scores across different architectures, ZODE converts raw model outputs into p-values based on a small validation set of in-distribution (ID) data.
  3. Sample-Aware Selection via Benjamini-Hochberg: To avoid the high false-positive rates associated with naive ensembles, ZODE employs the Benjamini-Hochberg (BH) procedure. This statistical method dynamically identifies "active" detectors for each specific test input.

Key Insights: The Power of Diversity

The success of ZODE stems from the inherent diversity of its model zoo. Different architectures perceive the world through different lenses; while one model might be sensitive to global shapes, another might focus on local textures.

Diverse OOD samples effectively detected by different models in the ZODE zoo.

By using p-values and the BH procedure, ZODE provides a statistically rigorous way to harness these diverse perspectives. It allows "specialist" models to flag specific OOD samples that a general-purpose model might miss, while maintaining a high True Positive Rate for in-distribution data.

Experimental Results

ZODE was evaluated on CIFAR10 and ImageNet-1K against a variety of OOD datasets (e.g., SVHN, LSUN, iNaturalist, SUN).

  • CIFAR10: ZODE reduced the average False Positive Rate (FPR) from 11.07% to 3.83%, representing a 65.4% relative improvement.
  • ImageNet: The average FPR dropped from 38.47% to 24.14%, a 37.25% relative improvement.

Crucially, ZODE remained robust across distribution shifts, adaptively switching to the most relevant models for each test case.

Conclusion and Future Work

ZODE demonstrates that a sample-aware ensemble of diverse models can significantly outperform single-model OOD detectors. While the computational cost of running a model zoo is higher, the gains in safety and reliability are substantial. Future research will focus on optimizing the zoo through model pruning and distributed computing to make ZODE more suitable for edge-device deployment.

Summary updated

Abstract

In this work, we present a novel perspective on detecting out-of-distribution (OOD) samples and propose an algorithm for sample-aware model selection to enhance the effectiveness of OOD detection. Our algorithm determines, for each test input, which pre-trained models in the model zoo are capable of identifying the test input as an OOD sample. If no such models exist in the model zoo, the test input is classified as an in-distribution (ID) sample. We the-oretically demonstrate that our method maintains the true positive rate of ID samples and accurately identifies OOD samples with high probability when there are a sufficient number of diverse pre-trained models in the model zoo. Extensive experiments were conducted to validate our method, demonstrating that it leverages the complementarity among single-model detectors to consistently improve the effective-ness of OOD sample identification. Compared to baseline methods, our approach improved the relative performance by 65.40% and 37.25% on the CIFAR10 and ImageNet benchmarks, respectively.
BibTeX
@inproceedings{xue2024enhancing,
  author = {Feng Xue and Zi He and Yuan Zhang and Chuanlong Xie and Zhenguo Li and Falong Tan},
  title = {{Enhancing the Power of OOD Detection via Sample-Aware Model Selection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01623},
}
Figure