On the Effectiveness of Sparsification for Detecting the Deep Unknowns

Yiyou Sun, Yixuan Li

Abstract

Detecting out-of-distribution (OOD) inputs is a central challenge for safely deploying machine learning models in the real world. Previous methods commonly rely on an OOD score derived from the overparameterized weight space, while largely overlooking the role of sparsification. In this paper, we reveal important insights that reliance on unimportant weights and units can directly attribute to the brittleness of OOD detection. To mitigate the issue, we propose a sparsification-based OOD detection framework termed DICE. Our key idea is to rank weights based on a measure of contribution, and selectively use the most salient weights to derive the output for OOD detection. We provide both empirical and theoretical insights, characterizing and explaining the mechanism by which DICE improves OOD detection. By pruning away noisy signals, DICE provably reduces the output variance for OOD data, resulting in a sharper output distribution and stronger separability from ID data. We demonstrate the effectiveness of sparsification-based OOD detection on several benchmarks and establish competitive performance.

Introduction

Deep neural networks are notoriously overconfident. When encountering an input from a class they have never seen—known as out-of-distribution (OOD) data—they often assign it a high-confidence score. This "brittleness" is a major safety hurdle for deploying machine learning in critical real-world applications, such as autonomous driving or medical diagnosis.

The authors identify overparameterization as a key culprit. Modern networks contain millions of weights. While many are crucial for identifying in-distribution (ID) classes, others are "unimportant" for those tasks. However, OOD inputs can trigger these unimportant units, creating "noisy signals" that aggregate into the final output. This noise increases the variance of the model's predictions for OOD data, causing their score distributions to overlap with ID data and making them difficult to filter out reliably.

Methodology: Directed Sparsification (DICE)

To address this, the authors propose DICE (Directed Sparsification), a post hoc framework designed specifically to sharpen OOD detection. Unlike traditional pruning for hardware efficiency, DICE selectively uses only the most salient weights to derive the output for OOD detection.

The process involves three main steps:

  1. Measure Contribution: For each class, a "contribution matrix" is calculated based on the element-wise product of weights and activations, averaged over a small set of ID training data.
  2. Rank and Mask: Weights are ranked by their contribution. DICE creates a binary mask to keep only the top-$k$ most contributing weights (the "major features") and sets the rest to zero.
  3. Energy-Based Scoring: During inference, the model uses this sparsified weight matrix to derive an energy score.

Because DICE is applied post hoc to the final layer of a pre-trained network, it preserves the model's original classification accuracy on ID data while significantly improving its ability to flag OOD samples.

Key Insights: The Variance Reduction Effect

The core theoretical insight is that pruning acts as a variance reduction technique. By removing the "noise floor" created by unimportant weights, DICE results in a much sharper score distribution for OOD data.

Sparsification via DICE reduces the output variance for OOD data, leading to a sharper distribution and better separation from ID data.

When the variance of OOD scores is reduced, the overlap between ID and OOD distributions shrinks, making thresholding much more effective. This reinforces the "familiarity hypothesis": OOD detection succeeds by identifying "familiar features." By forcing the model to rely only on the weights that define its training classes, it becomes much more sensitive to inputs that lack those specific patterns.

Experimental Results

The researchers evaluated DICE across several standard and large-scale benchmarks, including CIFAR-10, CIFAR-100, and ImageNet-1k, against a diverse suite of OOD datasets such as SVHN, Places365, and iNaturalist.

DICE establishes a new state-of-the-art for post hoc OOD detection:

  • CIFAR Benchmarks: On CIFAR-100, DICE reduces the average False Positive Rate (FPR95) to 49.72%, a significant improvement over the standard Energy score (68.45%) and the competitive ReAct baseline (62.27%).
  • Large-Scale ImageNet: On ImageNet, DICE (combined with ReAct) achieves an average FPR95 of 27.25%, outperforming all other post hoc methods.
Performance comparison of DICE against competitive baselines on CIFAR-10 and CIFAR-100 benchmarks.

Furthermore, DICE is highly efficient; because it only requires a simple weight mask, it adds negligible computational overhead during inference.

Conclusion

DICE demonstrates that sparsification is a powerful tool for robust OOD detection. By pruning unimportant weights, it reduces output variance and improves the separability of unknown inputs. While the optimal sparsity level can vary, the method provides a robust validation strategy. Future work may explore applying directed sparsification to deeper layers or integrating these objectives directly into the training process to create inherently more robust models.

Summary updated

Abstract

Detecting out-of-distribution (OOD) inputs is a central challenge for safely deploying machine learning models in the real world. Previous methods commonly rely on an OOD score derived from the overparameterized weight space, while largely overlooking the role of sparsification. In this paper, we reveal important insights that reliance on unimportant weights and units can directly attribute to the brittleness of OOD detection. To mitigate the issue, we propose a sparsification-based OOD detection framework termed DICE. Our key idea is to rank weights based on a measure of contribution, and selectively use the most salient weights to derive the output for OOD detection. We provide both empirical and theoretical insights, characterizing and explaining the mechanism by which DICE improves OOD detection. By pruning away noisy signals, DICE provably reduces the output variance for OOD data, resulting in a sharper output distribution and stronger separability from ID data. We demonstrate the effectiveness of sparsification-based OOD detection on several benchmarks and establish competitive performance.
BibTeX
@inproceedings{sun2022effectiveness,
  author = {Yiyou Sun and Yixuan Li},
  title = {{On the Effectiveness of Sparsification for Detecting the Deep Unknowns}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20053-3\_40},
}
Figure