A Robust Out-of-Distribution Detection Framework via Synergistic Smoothing

Maria Stoica, Abdelrahman Hekal, Alessio Lomuscio

TL;DR

Out-of-Distribution (OOD) detectors are crucial for safely deploying machine learning models, yet they are highly vulnerable to adversarial attacks. This paper introduces ROSS, a post-hoc, gradient-free framework that uses median smoothing and score instability (Median Absolute Deviation) to provide symmetric robustness against both score-maximizing and score-minimizing attacks, outperforming prior methods by up to 40 AUROC points.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are notorious for making highly confident but completely incorrect predictions when faced with data outside their training distribution (Out-of-Distribution, or OOD data). While researchers have developed OOD detectors to flag these unsafe inputs, recent work shows that these detectors themselves can be easily fooled by adversarial perturbations.

Existing defenses that use input perturbations (like ODIN or PRO) suffer from an asymmetric vulnerability. Because they rely on directional, gradient-based optimization to either artificially boost or reduce model confidence, they successfully defend against one type of attack direction but remain completely blind to the opposite one (e.g., defending against score-minimizing attacks while failing against score-maximizing attacks). The authors address this gap by proposing a defense that is robust against attacks from any direction without requiring expensive model retraining.

Methods

The authors introduce ROSS (Robust OOD Detector via Synergistic Smoothing), a model-agnostic, post-hoc framework that can be applied to any standard OOD scoring function (such as MSP, Energy, or GEN).

Instead of using gradients to perturb the input in a specific direction, ROSS generates $N$ noisy variants of an input by adding random Gaussian noise. It passes these variants through the base OOD detector to create a "score stack" and extracts two robust statistical metrics:

  1. Median Score ($S_{med}$): Represents the robust central tendency of the smoothed OOD score.
  2. Score Stability ($\omega_{med}$): Measured via the Median Absolute Deviation (MAD), this quantifies the local instability of the score landscape.

Crucially, ROSS employs a confidence-thresholded scoring function. It only applies a stability bonus if the median score exceeds a threshold ($S_{95}$, defined as the 5th percentile of the validation set). This gating mechanism ensures that "far-OOD" samples—which might be highly stable but consistently low-confidence—are not mistakenly rewarded and misclassified as in-distribution (ID).

Datasets

The framework was evaluated using the standardized protocol from the OpenOOD benchmark:

  • In-Distribution (ID) Datasets: CIFAR-10, CIFAR-100, and ImageNet.
  • Out-of-Distribution (OOD) Datasets: Divided into near-OOD (CIFAR-100, TinyImageNet) and far-OOD (MNIST, SVHN, Texture, Places365).

Key Insights and Ideas

  • Instability as an OOD Signal: ID samples typically sit in smooth, well-defined regions of the model's score landscape, meaning small random perturbations won't change their scores much. OOD samples inhabit uncharted territory where the model behaves erratically, leading to high score variance (instability) under the exact same perturbations.
Figure 1. Comparison of GEN score profiles for an ID sample (CIFAR-10) and an OOD sample (Tiny ImageNet) under input per- turbations (25 noisy variants). Despite similar medians, the OOD sample shows much higher instability (MAD), illustrating how in- stability metrics help distinguish ID from OOD when base scores overlap.
  • The Danger of Gradients in Defenses: Relying on gradient-based, directional optimization to calculate OOD scores creates a fundamental blind spot. By switching to a gradient-free, non-directional approach using random perturbations, ROSS achieves symmetric robustness against both score-minimizing and score-maximizing attacks.
  • The Necessity of Gating: Simply rewarding score stability across the board fails because some completely unrelated "far-OOD" inputs produce highly stable, flat, but low-confidence score landscapes due to network indifference. Gating via the $S_{95}$ threshold ensures stability only benefits plausible ID candidates.

Key Results

  • Symmetric Adversarial Robustness: Under severe adversarial attacks (PGD-Max, $\epsilon = 8/255$) on CIFAR-10, the baseline Maximum Softmax Probability (MSP) detector's performance collapses to an AUROC of 3.76%. Applying ROSS (ROSS-MSP) recovers this performance substantially to 52.65%.
  • Outperforming Prior Perturbation Methods: On the CIFAR-100 benchmark under PGD-Max attacks ($\epsilon = 4/255$), the prior state-of-the-art perturbation method, PRO-fDBD, actually degrades the baseline performance down to 28.38% AUROC due to its directional nature. ROSS-fDBD avoids this trap, achieving 58.48% AUROC—representing a massive 30+ point absolute improvement.
  • Complexity Amplifies Gains: The performance improvements delivered by ROSS were significantly more pronounced on CIFAR-100 than on CIFAR-10, indicating that ROSS is particularly effective at safeguarding the complex, high-dimensional decision boundaries found in many-class problems.

Limitations and Future Work

  • Not a Complete Substitute for Adversarial Training: While ROSS provides an easily deployable, lightweight post-hoc first line of defense, extremely sophisticated or intense attacks may still require computationally heavy adversarial training methods (like ALOE or ATOM).
  • Scope of Attacks: The evaluation primarily focused on gradient-based adversarial attacks. The authors note that future work should explore extending ROSS to handle broader distribution shifts, such as semantic perturbations, real-world environmental corruptions, and shifts induced by generative models.

Summary updated

Abstract

Reliable out-of-distribution (OOD) detection is a critical requirement for the safe deployment of machine learning systems. Despite recent progress, state-of-the-art OOD detectors are highly susceptible to adversarial attacks, which undermines their trustworthiness in automated systems. To address this vulnerability, we apply median smoothing to baseline OOD detection scores, balancing clean and adversarial accuracies. Our key insight is that the noisy samples generated for median smoothing can be repurposed to quantify the local instability of the base score. We observe that OOD samples exhibit higher instability under perturbation. Based on this, we propose ROSS, a novel and robust post-hoc OOD detector that leverages the instability of baseline scores to further distinguish between in-distribution (ID) and OOD samples. ROSS achieves symmetric robustness, performing strongly against both score-minimising and score-maximising attacks, unlike prior work. This symmetric defence leads to state-of-the-art robustness, outperforming prior methods by up to 40 AUROC points. We demonstrate ROSS's effectiveness on extensive experiments across CIFAR-10, CIFAR-100, and ImageNet. Code is available at: https://github.com/Abdu-Hekal/ROSS.
BibTeX
@inproceedings{stoica2026robust,
  author = {Maria Stoica and Abdelrahman Hekal and Alessio Lomuscio},
  title = {{A Robust Out-of-Distribution Detection Framework via Synergistic Smoothing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=LRDf0qSxma},
}

Loading related content...

Figure