Dynamic Sliced Wasserstein Approximation with Decaying Projection Kernels for Knowledge Distillation

Hongyun Lu, Zhi Liu, Mengmeng Zhang, Yuan Li, Ran Cheng, Jianfeng QuIEICE 2025 · DOI

Abstract

Knowledge distillation (KD) is effective for compressing object detection models into lightweight student models. However, traditional distillation losses often fail to capture the geometric structure of high-dimensional feature distributions. In this paper, we propose a Dynamic Sliced Wasserstein Approximation (DSWA) distillation framework for YOLOv8-based object detection, leveraging the Sliced Wasserstein Distance (SWD) with adaptive projection reduction. Our key innovation lies in dynamically decreasing the number of random projections during training. Starting with dense projections for accurate feature alignment, DSWA progressively sparsifies projections to suppress noise and accelerate convergence. Experiments on the BDD100K dataset demonstrate that DSWA achieves a 2.8% mAP improvement over YOLOv8. Compared to fixed-length projection SWD variants, our method reduces training time by 32%.

Introduction

Knowledge distillation (KD) has emerged as a cornerstone technique for deploying high-performance object detection models on resource-constrained devices. By transferring knowledge from a complex "teacher" model to a compact "student" model, KD aims to retain predictive accuracy while significantly reducing computational overhead. In the context of modern architectures like YOLOv8, effective distillation typically involves aligning the intermediate feature maps of the two models.

However, standard alignment metrics such as Mean Squared Error (MSE) or Kullback-Leibler (KL) divergence often fall short. These methods treat feature distributions as collections of independent points, neglecting the underlying geometric structure of the high-dimensional feature space. The Sliced Wasserstein Distance (SWD) offers a theoretically grounded alternative by capturing these geometric relationships through 1D projections. Despite its advantages, the computational cost of SWD scales with the number of random projections required for accurate estimation, often leading to prohibitive training times or unstable gradients if the projection count is too low.

Methodology: Dynamic Sliced Wasserstein Approximation (DSWA)

To address the efficiency-accuracy trade-off in SWD-based distillation, we introduce the Dynamic Sliced Wasserstein Approximation (DSWA) framework. DSWA optimizes the distillation process by varying the complexity of the loss function over the course of training.

Decaying Projection Kernels

The central mechanism of DSWA is the use of decaying projection kernels. Rather than maintaining a static number of random projections ($K$), we implement a stepwise linear annealing schedule.

Overview of the DSWA framework, illustrating the decaying projection kernels from dense (512) to sparse (64) states during training.

As illustrated in the figure above, the training process is divided into stages:

  1. Initial Alignment (Dense State): Training commences with a high number of projections (e.g., $K=512$). This provides the student model with a high-fidelity gradient, ensuring precise initial alignment with the teacher's feature distribution.
  2. Progressive Sparsification: The number of projections is reduced in a stepwise fashion (e.g., decreasing by 64 every few epochs).
  3. Convergence (Sparse State): By the final stages of training, the model utilizes a minimal number of projections (e.g., $K=64$). This reduction not only accelerates computation but also serves as a form of regularization, helping to suppress noise once the primary feature structures have been learned.

Stepwise vs. Continuous Decay

Our empirical findings suggest that a stepwise linear decay is superior to continuous schedules like cosine decay. Maintaining a constant projection count for a fixed interval ($\tau$) allows the model's representational capacity to stabilize before transitioning to a lower dimensionality, leading to more robust convergence.

Experimental Results

We evaluated DSWA using a YOLOv8m teacher and a YOLOv8s student on the BDD100K and KITTI autonomous driving benchmarks.

Performance on BDD100K

On the BDD100K dataset, which features diverse and challenging traffic scenarios, the DSWA-distilled YOLOv8s model achieved a mean Average Precision (mAP) of 30.14%. This represents a substantial 2.8% improvement over the baseline YOLOv8s model (27.33%).

Computational Efficiency

The primary advantage of DSWA is its impact on training throughput. By dynamically reducing the projection count, we achieved a 32% reduction in total training time compared to a fixed-projection SWD approach ($K=512$). Specifically, training time dropped from 7.49 hours to 5.09 hours on our experimental hardware, without a significant loss in final model accuracy.

Conclusion

The Dynamic Sliced Wasserstein Approximation (DSWA) framework provides a scalable and efficient solution for knowledge distillation in object detection. By leveraging a "coarse-to-fine" strategy via decaying projection kernels, DSWA captures the geometric nuances of feature distributions while drastically reducing the computational burden of Wasserstein-based losses. Future research will focus on the sensitivity of the decay interval ($\tau$) and the applicability of DSWA to transformer-based architectures and other vision tasks such as semantic segmentation.

Summary updated

Abstract

Knowledge distillation (KD) is effective for compressing object detection models into lightweight student models. However, traditional distillation losses often fail to capture the geometric structure of high-dimensional feature distributions. In this paper, we propose a Dynamic Sliced Wasserstein Approximation (DSWA) distillation framework for YOLOv8-based object detection, leveraging the Sliced Wasserstein Distance (SWD) with adaptive projection reduction. Our key innovation lies in dynamically decreasing the number of random projections during training. Starting with dense projections for accurate feature alignment, DSWA progressively sparsifies projections to suppress noise and accelerate convergence. Experiments on the BDD100K dataset demonstrate that DSWA achieves a 2.8% mAP improvement over YOLOv8. Compared to fixed-length projection SWD variants, our method reduces training time by 32%.
BibTeX
@inproceedings{lu2025dynamic,
  author = {Hongyun Lu and Zhi Liu and Mengmeng Zhang and Yuan Li and Ran Cheng and Jianfeng Qu},
  title = {{Dynamic Sliced Wasserstein Approximation with Decaying Projection Kernels for Knowledge Distillation}},
  booktitle = {IEICE},
  year = {2025},
  doi = {10.1587/transfun.2025sml0001},
}
Figure