Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation

Yang Yang, Haonan XuICML 2025 · OpenReview

In the quest to build reliable AI, Out-of-Distribution (OOD) detection is a critical safety net. We want our models to not only be accurate on what they know but also humble enough to say "I don't know" when they encounter something completely new.

A common frustration in training these models is that as they get better at classifying their training data, they often get worse at OOD detection. A new paper from researchers at the Nanjing University of Science and Technology, titled "Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation," proposes a clever way to break this trade-off.

TL;DR

Instead of discarding "atypical" training samples that cause models to become overconfident, the Progressive Self-Knowledge Distillation (PSKD) framework uses them as teaching moments. By distilling uncertainty-rich "soft targets" from a previous, better-calibrated version of itself, the model improves both its general accuracy and its ability to spot OOD inputs.

What Problem Does This Paper Address?

Deep learning models are notorious for "overfitting" to the training distribution in a specific way: they eventually memorize atypical samples—those weird, noisy, or hard-to-classify images that sit at the fringes of the data.

The trade-off between ID accuracy and OOD detection during training, and how different methods handle atypical samples.

While memorizing these helps the model squeeze out a few extra points of training accuracy, it has a nasty side effect. It forces the model to create sharp, confident decision boundaries even in regions where it should be uncertain. Consequently, when the model sees an OOD image (like a picture of a tractor when it was only trained on cats and dogs), it classifies it with high confidence.

Previous research suggested "forgetting" these atypical samples via gradient ascent. However, the authors of this paper argue that this is a waste of perfectly good data. Forgetting atypical samples hurts the model's general performance (ID accuracy) and limits its potential. Instead, the authors propose learning from them in a smarter way.

Methods: Progressive Self-Knowledge Distillation (PSKD)

The authors propose a "self-learning" paradigm. If the model was better at OOD detection at an earlier stage of training, why not let that "younger, wiser" version teach the current version?

Overview of the Progressive Self-Knowledge Distillation (PSKD) framework, showing the interaction between the Teacher and Student models.
  1. Self-Selection of the Teacher: The framework periodically evaluates the model's OOD detection capability using pseudo-outliers (synthetic OOD data created by distorting training images). The version of the model that achieves the best AUROC (Area Under the Receiver Operating Characteristic) is saved as the "Teacher."
  2. Multi-Level Distillation: The current "Student" model is trained using a combination of standard labels and "hints" from the Teacher.
  3. Response-level: The student tries to match the Teacher’s output probabilities (logits). Since the Teacher is less overconfident, these "soft targets" embed valuable uncertainty information.
  4. Feature-level: The student also tries to mimic the Teacher’s internal feature representations (the penultimate layer).
  5. Progressive Weighting: The influence of the Teacher increases over time using a cosine annealing schedule. Early in training, the model focuses on learning the basics; later on, the distillation kicks in to prevent the overconfidence that usually comes with late-stage memorization.

Datasets

The authors conducted extensive testing across several standard benchmarks:

  • In-Distribution (ID): CIFAR-10, CIFAR-100, and a large-scale ImageNet-200 subset.
  • Out-of-Distribution (OOD):
    • Near-OOD: CIFAR-100 (when ID is CIFAR-10), TinyImageNet, SSB-hard, and NINCO.
  • Far-OOD: MNIST, SVHN, Textures, and Places365.
  • Pseudo-Outliers: Created using rotation, Gaussian noise, and image distortion for the self-selection process.

Key Insights and Ideas

  • The "OOD Peak": The paper confirms that a model's OOD detection capability usually peaks before it fully converges on the training task. PSKD effectively "freezes" that peak performance and distills it into the final model.
  • Atypical Samples are Lessons, not Noise: By using soft targets, atypical samples teach the model where to be uncertain. This is more effective than simply deleting them from the training set.
  • Orthogonality: PSKD isn't a standalone scoring function; it’s a training-time "plugin." It can be combined with existing methods like Energy-based scores or Outlier Exposure (OE) to make them even better.
  • Restoring Intrinsic Capability: The authors show that vanilla training actually suppresses a model's natural OOD detection potential. PSKD "restores" this intrinsic capability.

Key Results

The results show a "dual success" in both ID accuracy and OOD detection:

  • CIFAR-10: Combining Energy-based detection with PSKD reduced the FPR95 (False Positive Rate at 95% True Positive Rate) from 48.17% to 31.67% for Near-OOD tasks, while actually increasing ID accuracy.
  • ImageNet-200: On the more challenging large-scale benchmark, PSKD consistently outperformed "Unleashing Mask" (the forgetting-based method), maintaining higher ID accuracy while providing superior OOD separation.
  • Robustness: The method proved effective across different architectures, including ResNet-18 and Vision Transformers (ViT).

Limitations and Future Work

  • Pseudo-Outlier Quality: The effectiveness of the self-teacher selection depends on how well the pseudo-outliers (distortions/noise) represent real-world OOD data. While the authors show it works well, there is always a gap between synthetic and real outliers.
  • Computational Overhead: While the authors argue the overhead is minimal (adding only a few minutes to a 400-minute ImageNet training run), it does require periodically saving and running inference on teacher models.
  • Hyperparameter Tuning: The method introduces new parameters (like the distillation weight $\alpha$ and temperature $T$), which might require tuning for specific new architectures or niche datasets.

Overall, PSKD offers a more sophisticated philosophy for robust AI: don't just ignore the hard parts of your data—use them to learn the limits of your own knowledge.

Summary updated

Abstract

Out-of-distribution (OOD) detection aims to ensure AI system reliability by rejecting inputs outside the training distribution. Recent work shows that memorizing atypical samples during later stages of training can hurt OOD detection, while strategies for forgetting them show promising improvements. However, directly forgetting atypical samples sacrifices ID generalization and limits the model's OOD detection capability. To address this issue, we propose Progressive Self-Knowledge Distillation (PSKD) framework, which strengthens the OOD detection capability by leveraging self-provided uncertainty-embedded targets. Specifically, PSKD adaptively selects a self-teacher model from the training history using pseudo-outliers, facilitating the learning of uncertainty knowledge via multi-level distillation applied to features and responses. As a result, PSKD achieves better ID generalization and uncertainty estimation for OOD detection. Moreover, PSKD is orthogonal to most existing methods and can be integrated as a plugin to collaborate with them. Experimental results from multiple OOD scenarios verify the effectiveness and general applicability of PSKD.
BibTeX
@inproceedings{yang2025strengthen,
  author = {Yang Yang and Haonan Xu},
  title = {{Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=iA8lD3oG5G},
}
Figure