Out-of-Distribution Detection Is Not All You Need

Joris Guérin, Kevin Delmas, Raul Sena Ferreira, Jérémie GuiochetAAAI 2023 · DOI

Introduction: The OOD Safety Gap

In the pursuit of safe AI deployment, researchers have long relied on Out-of-Distribution (OOD) detection as the primary benchmark for runtime monitors. The logic is intuitive: if a model encounters data significantly different from its training set, it should be flagged as unsafe. However, this paper, presented at AAAI 2023, argues that OOD detection is a flawed proxy for actual safety.

Runtime monitoring: the monitor evaluates new input data to decide whether to accept or reject the neural network's prediction.

The authors identify two critical gaps in the OOD paradigm. First, model generalization means a network might correctly classify OOD samples; rejecting these unnecessarily reduces system availability. Second, and more dangerously, in-distribution errors occur when a model fails on data that looks exactly like its training set. Standard OOD monitors are blind to these failures, creating a dangerous false sense of security in safety-critical applications like autonomous driving or medical robotics.

Beyond OOD: Out-of-Model-Scope (OMS) Detection

To address these gaps, the authors formalize the Out-of-Model-Scope (OMS) detection task. Unlike OOD, which focuses on data novelty, OMS focuses on model reliability. The goal is to detect when a specific model $f$ is likely to produce an incorrect prediction, regardless of whether the input is "new" or "familiar." They mathematically define the "Model Scope" ($D_f$) as the set of instances where the model is accurate, shifting the monitor's objective from finding "weird data" to finding "model-breaking data."

Methodology: The "Clean Training" Trick

The paper evaluates six popular monitoring methods (including MSP, Energy-based, and Mahalanobis distance) across 54 diverse scenarios. A key contribution is a simple yet effective "training trick" to improve monitor performance.

Monitor training phase: fitting a model of 'safe' data instances using the neural network and training dataset.

Most monitors are typically trained on the entire training dataset. The authors propose "clean" training: removing the small fraction of samples (often <1%) that the neural network misclassified during its own training phase. By fitting the monitor only on correctly classified data, the decision boundaries are not "stretched" by training-set outliers, leading to a more precise definition of the model's safe operating envelope.

Experimental Evaluation and Results

The study utilized a comprehensive suite of datasets (CIFAR, SVHN) and perturbations, including covariate shifts (brightness, blur) and adversarial attacks.

Key Insights:

  • OOD Ambiguity: OOD is a moving target with no objective threshold, making it unreliable for consistent safety guarantees.
  • The "Perfect" OOD Trap: Even a theoretically perfect OOD detector would miss over 7% of model errors in novelty tasks and significantly more during covariate shifts.
  • Misleading Rankings: Monitors that excel in OOD benchmarks often perform poorly in OMS tasks. Relying solely on OOD metrics can lead to the selection of suboptimal safety tools.
  • Effectiveness of Clean Training: Using the "Only Correct Data" trick for the Outside-the-Box (OtB) monitor improved mean recall from 0.75 to 0.77 across all 54 scenarios—a statistically significant gain (p-value 2.6e-9) without architectural complexity.

Conclusion

The transition from OOD to OMS detection represents a necessary shift toward model-centric safety. By focusing on the actual boundaries of a model's competence rather than just data distribution, we can build more robust runtime monitors. The "clean training" trick demonstrates that significant safety gains can be achieved through better data curation for monitors, paving the way for more reliable AI systems in the real world.

Summary updated

BibTeX
@inproceedings{guerin2023out,
  author = {Joris Guérin and Kevin Delmas and Raul Sena Ferreira and Jérémie Guiochet},
  title = {{Out-of-Distribution Detection Is Not All You Need}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26732},
}
Figure