Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation

Kailin Lyu, Zherui Zhang, Junhao Dong, Kexue Fu, Weiguang Pang, Rongtao Xu, Qizheng Wang, Di Wu, Chee-Keong Kwoh, Longxiang Gao, Shibiao Xu, Changwei Wang, Ce Hao, Yu ZhangarXiv 2026 · arXiv

TL;DR

Data-Free Knowledge Distillation (DFKD) allows a student network to learn from a teacher without accessing the original training data, but generators often "cheat" by exploiting frequency shortcuts, leading to poor image diversity and unstable training. This paper introduces CSWL (Close Shortcut Wins Long), a framework that uses frequency-domain feature augmentation and a cross-stage reconstruction task to force the generator to produce highly diverse, stable images. The result is a significant boost in student accuracy and generalization across multiple vision benchmarks.

What Problem Does This Paper Address?

Knowledge distillation is great for compressing large models, but when the original training data is unavailable due to privacy or storage constraints, we turn to Data-Free Knowledge Distillation (DFKD). Typical generator-based DFKD methods train a generator adversarially against a fixed teacher network to synthesize training images from scratch.

However, these generators suffer from generative shortcut learning. Instead of learning the true underlying data distribution, they over-rely on specific frequency components and positions that the teacher model prefers. This causes a massive imbalance: a few "preferred" classes look semantically clear, while "weaker" classes remain abstract, noisy blobs.

Fig. 1. Synthetic images were sampled from the mid-training phase of the generator (90 to 100 epochs). Panels (a) and (b) illustrate the preferred categories, “Bird” and “Frog”, which demonstrate relatively clear semantic content, whereas panel (c) displays images of weaker categories, characterized by abstract and noisy features.

Furthermore, attempts to fix this diversity gap in the spatial domain often introduce severe training instability, causing metrics like the Inception Score (IS) to fluctuate wildly.

Methods

The authors propose the CSWL framework to tackle shortcut learning and instability directly in the frequency domain using two core components:

  1. Class-Decoupled Frequency Augmentation (CDFA): Instead of manipulating pixels directly, CDFA applies a Fast Fourier Transform (FFT) to the generator's intermediate features, decomposing them into amplitude (which captures style and energy) and phase (which preserves structural semantics). It then performs intra-class stochastic amplitude scaling and inter-class amplitude mixing. This breaks the generator's hyper-fixation on specific frequencies and injects diverse class characteristics without destroying the image's layout.

  2. Cross-Stage Frequency Reconstruction (CSFR): To stabilize training, the authors introduce an auxiliary task. It uses an amplitude-guided mask to hide the most salient frequency components (like edges and textures) of a synthetic image. A decoder—shared between the data generation stage (using the teacher as an encoder) and the knowledge transfer stage (using the student as an encoder)—is tasked with reconstructing the original image. This shared decoder acts as a powerful regularizer, smoothing out training fluctuations.

Datasets

The framework was validated across several standard image recognition datasets spanning multiple resolutions:

  • Low Resolution (32×32): CIFAR-10 and CIFAR-100
  • Medium Resolution (64×64): Tiny-ImageNet
  • High Resolution (224×224): ImageNet-1K
  • Downstream Transfer: NYUv2 (used to evaluate semantic segmentation performance)

Key Insights and Ideas

  • The Frequency Domain is a Cleaner Lens: Spatial-domain strategies for diversity can be abstract and hard to interpret. Looking at DFKD through Fourier analysis reveals that weakly predicted classes rely far more on high-frequency components than preferred classes do.
  • The Diversity-Stability Paradox: There is an inherent conflict in DFKD between the flat distribution required to generate diverse classes and the sharp distribution required for effective knowledge transfer. CDFA fixes the distribution flatness, but CSFR is required to bridge the optimization gap and stabilize the generator.
  • Implicit EMA via Shared Decoders: By sharing the reconstruction decoder across both the generation and distillation stages, the network implicitly creates an Exponential Moving Average (EMA) effect, transferring stable reconstruction capabilities back to the generation phase.

Key Results

CSWL consistently outperformed state-of-the-art DFKD methods across various network architectures (ResNet, VGG, WideResNet):

  • Accuracy Gains: On CIFAR-100 (with a WRN-40-2 teacher and WRN-16-1 student), CSWL achieved a 71.85% accuracy, outperforming the previous best spatial method (SpaceshipNet) by 1.90%.
  • Downstream Transfer: When evaluated on the NYUv2 semantic segmentation task using DeepLabV3, CSWL improved the mean Intersection over Union (mIoU) by 3.03% over competing data-free methods.
  • Data-Free Quantization: CSWL also proved highly effective when adapted for data-free network quantization, beating methods like ZeroQ and HAST under ultra-low bit-width settings (W3A3).
  • Training Stability: As shown below, enabling the CSFR module successfully dampens the wild Inception Score fluctuations common in standard generator training.
Inception Score (IS) variation curves showing stabilized generator training when CSFR is enabled.

Limitations and Future Work

While CSWL shows strong performance on standard classification benchmarks, the authors note that it is primarily optimized for standard network architectures. Future work will focus on extending the framework to support more complex, heterogeneous network architectures. Additionally, they plan to explore its applicability in more complex vision tasks beyond classification and segmentation, such as object detection, instance segmentation, and multimodal learning scenarios.

Summary updated

Abstract

Data-Free Knowledge Distillation (DFKD) preserves privacy by transferring knowledge without real data access. However, existing generator-based DFKD methods suffer from over-reliance on teacher preferences and pattern collapse, exhibiting "generative shortcut learning" in the frequency domain: dependent on specific frequency components and frequency positions, resulting in inconsistent synthetic image quality and class diversity. In this paper, we propose a CSWL framework aimed at introducing insights from the frequency domain perspective to improve generator diversity and training stability to Close the phenomenon of Shortcut learning to Win in the Longer term. To address the issue of generative shortcut learning, we introduce frequency-domain augmentation at the feature level, encouraging the generator to attend to the full frequency spectrum and thereby suppress shortcut learning behavior. To tackle training instability, we propose a Cross-Stage Frequency Reconstruction (CSFR) auxiliary task, which implicitly constructs an Exponential Moving Average (EMA) mechanism to promote long-term optimization and stability. Extensive experiments, including downstream tasks and various image recognition datasets at multiple resolutions, validate the effectiveness of CSWL in improving both diversity and stability from the frequency view.
BibTeX
@misc{lyu2026close,
  author = {Kailin Lyu and Zherui Zhang and Junhao Dong and Kexue Fu and Weiguang Pang and Rongtao Xu and Qizheng Wang and Di Wu and Chee-Keong Kwoh and Longxiang Gao and Shibiao Xu and Changwei Wang and Ce Hao and Yu Zhang},
  title = {{Close Shortcut Wins Long: Seeking Diverse and Stable Generators for Data-Free Knowledge Distillation}},
  year = {2026},
  eprint = {2608.22003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.22003},
}
Figure