SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization

Tongtong Su, Yun Liao, Fengbo ZhengNeurIPS 2025 · OpenReview

TL;DR

SDPGO (Self-Distillation via Proximal Gradient Optimization) is a framework that allows a neural network to act as its own teacher more effectively by using real-time gradient information to identify "task-critical" features. By prioritizing high-impact features and refining knowledge through sequential iterations, it achieves state-of-the-art performance across classification, detection, and segmentation tasks without the computational overhead of a separate teacher model.

What Problem Does This Paper Address?

Knowledge Distillation (KD) is the standard way to "compress" a large, highly capable model (the teacher) into a smaller, faster one (the student). However, traditional KD has two major drawbacks: it requires training a massive teacher model first, and there is often a "capacity gap" where the student simply cannot keep up with the teacher's complex logic.

Self-Knowledge Distillation (SKD) attempts to solve this by letting a single model learn from itself. Unfortunately, existing SKD methods have their own flaws. They often rely on fixed, hand-tuned weights to decide which features are important, or they use "auxiliary architectures" (extra branches on the network) that significantly increase memory usage. Most importantly, they tend to treat all features with equal importance, failing to adapt as the model evolves during training.

Comparison of Self-Knowledge Distillation paradigms, highlighting the proposed gradient optimization approach.

SDPGO aims to make SKD "smarter" by dynamically identifying which features actually drive performance at any given moment, eliminating the need for bulky auxiliary branches.

Methods

The authors introduce two primary modules that work in tandem to optimize the self-distillation process:

The overall architecture of the SDPGO framework, featuring the Sequential Iterative Learning (SIL) and Proximally Weight Assignment (PWA) modules.
  1. Sequential Iterative Learning (SIL) Module: Instead of waiting for a full epoch to finish to update the "teacher" signal, SIL partitions mini-batches into sequential segments. It calculates the KL divergence between the current predictions and those from the immediately preceding iteration. This creates a tight feedback loop that stabilizes training and allows the model to incrementally correct its own errors.
  2. Proximally Weight Assignment (PWA) Module: This is the core innovation. The framework calculates the gradients of intermediate features with respect to the task loss.
  3. The Logic: If a feature has a high gradient magnitude, it significantly impacts the final result.
  4. The "Proximal" Part: Borrowing from proximal gradient theory, the authors apply a soft-thresholding operator to these gradients. This suppresses "noisy" or low-impact features (setting their weights to zero) while amplifying the weights of critical features.
  5. Dynamic Weight Adjustment ($\alpha$): Unlike standard methods that use a fixed constant to balance the distillation loss and the primary task loss, SDPGO calculates a dynamic hyperparameter $\alpha$ based on the importance of features across all layers.

Key Insights and Ideas

  • Gradients as an Importance Map: The breakthrough here is treating gradients not just as a tool for optimization, but as a real-time diagnostic tool. Gradients tell the model exactly which parts of an image (or which internal representations) are currently most relevant to solving the task.
  • Sparsity is Strength: By using a proximal operator to enforce sparsity in feature weighting, the model avoids being distracted by "common" features that don't contribute to discriminative power. It forces the student to focus on the "hard" features that define the classes.
  • Cutting the Auxiliary Cord: Many state-of-the-art methods require adding extra "heads" or branches to the network during training. SDPGO achieves better results using the base architecture alone, making it much more memory-efficient.

Key Results

The researchers put SDPGO through a gauntlet of benchmarks, including general classification (CIFAR, ImageNet), fine-grained recognition (CUB200-2011, Cars196), and downstream tasks (MS-COCO, ADE20K). SDPGO consistently outperformed both traditional KD and recent SOTA self-distillation methods:

  • ImageNet: A ResNet-18 trained with SDPGO reached 72.47% top-1 accuracy, a significant 2.72% jump over the baseline and outperforming complex methods like FASD and MiSKD.
  • Efficiency: On a VGG-8 architecture, SDPGO achieved a 5x reduction in memory overhead compared to MixSKD while maintaining higher accuracy.
  • Edge Deployment: When tested on hardware like the Raspberry Pi 5, the method showed an average inference acceleration of 23.46% for lightweight backbones like ShuffleNet-V2, proving its utility for real-world mobile applications.
  • Robustness: The model proved significantly more resilient to noisy labels. At a 40% noise level on CIFAR-100, SDPGO maintained nearly 60% accuracy, while competitors dropped much further.

Limitations and Future Work

While the paper demonstrates strong results across CNNs and some ViT-like models (such as Swin-Transformer), the theoretical convergence proofs rely on standard convexity assumptions, which are rarely perfectly met in deep learning.

The authors suggest that future work could involve extending this gradient-driven weighting to Natural Language Processing (NLP) or Multimodal tasks, exploring how this method interacts with other optimization techniques like Sharpness-Aware Minimization (SAM), and further refining the proximal operator to handle even more complex data distributions.

Summary updated

Abstract

Self-knowledge distillation (SKD) enables single-model training by distilling knowledge from the model's own output, eliminating the need for a separate teacher network required in conventional distillation methods. However, current SKD methods focus mainly on replicating common features in the student model, neglecting the extraction of key features that significantly enhance student learning. Inspired by this, we devise a self-knowledge distillation framework entitled Self-Distillation training via Proximal Gradient Optimization or SDPGO, which utilizes gradient information to identify and assign greater weight to features that significantly impact classification performance, enabling the network to learn the most relevant features during training. Specifically, the proposed framework refines the gradient information into a dynamically changing weighting factor to evaluate the distillation knowledge via the dynamic weight adjustment scheme. Meanwhile, we devise the sequential iterative learning module to dynamically optimize knowledge transfer by leveraging historical predictions and real-time gradients, stabilizing training through mini-batch-based KL divergence refinement while adaptively prioritizing task-critical features for efficient self-distillation. Comprehensive experiments on image classification, object detection, and semantic segmentation demonstrate that our method consistently surpasses recent state-of-the-art knowledge distillation techniques. Code is available at: https://github.com/nanxiaotong/SDGPO.
BibTeX
@inproceedings{su2025sdpgo,
  author = {Tongtong Su and Yun Liao and Fengbo Zheng},
  title = {{SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=pbokMgz8e1},
}
Figure