Customizing Synthetic Data for Data-Free Student Learning

Shiya Luo, Defang Chen, Can Wang

Abstract

Knowledge Distillation (KD) typically requires access to original training data, which is often restricted due to privacy or proprietary concerns. Data-Free Knowledge Distillation (DFKD) addresses this by synthesizing data from a pre-trained teacher model. However, existing DFKD methods often generate samples that are either too simple for the student or rely on potentially inaccurate teacher predictions to define "hardness." This paper introduces Customizing Synthetic Data (CSD), a framework that adaptively generates "hard samples" tailored to the student's current learning state. By utilizing a self-supervised auxiliary task to measure student ability, CSD ensures synthetic data remains informative throughout training, significantly outperforming existing state-of-the-art methods.

The Challenge of Data-Free Distillation

Standard DFKD techniques often follow a teacher-centric adversarial approach. A generator is trained to produce samples that maximize the discrepancy between the teacher and student models. While intuitive, this approach has two major flaws:

  1. Teacher Misinformation: The teacher model was never trained on synthetic data. Its predictions on these out-of-distribution samples can be noisy or incorrect, leading the generator to produce samples that confuse rather than educate the student.
  2. Ignoring Student Progress: Static adversarial objectives do not account for the student's evolving understanding. Data that was "hard" at the start of training may become trivial later, leading to plateaued performance.

CSD shifts the focus from the teacher to the student, using the student's own performance on an auxiliary task to guide the generator.

A comparison between the traditional adversarial DFKD framework and the proposed student-centric CSD framework.

The CSD Framework

The CSD method operates through a two-stage iterative process: Adaptive Data Synthesis and Knowledge Transfer.

1. Adaptive Data Synthesis

Instead of relying solely on teacher-student disagreement, CSD introduces a self-supervised auxiliary task—image rotation prediction—to estimate student ability. The student is equipped with an auxiliary classifier tasked with predicting both the object category and the rotation angle (0°, 90°, 180°, or 270°) of a synthetic image.

The generator is trained to maximize the loss of this auxiliary task. This forces the generator to create samples that the student currently struggles to understand semantically. Because the rotation labels are known (ground truth), this process avoids the "teacher misinformation" trap. To maintain data realism, a "Narrowing the Distribution Gap" loss matches synthetic data statistics to the Batch Normalization (BN) layers of the pre-trained teacher.

2. Knowledge Transfer

Once hard samples are synthesized, the student learns from the teacher through a multi-faceted loss function:

  • KL Divergence: Aligning output probability distributions.
  • Feature Map Matching (MSE): Ensuring internal representations match the teacher's.
  • Cross-Entropy: Utilizing the labels assigned during the synthesis phase.
Detailed architecture of the CSD framework showing the forward and backward passes for both data synthesis and knowledge transfer.

Experimental Results

The researchers evaluated CSD across SVHN, CIFAR-10, and CIFAR-100 using various architecture pairs (e.g., ResNet, VGG, MobileNet).

Key Findings: * Superior Accuracy: CSD consistently outperformed representative methods like DAFL, ADI, and CMI. On the complex CIFAR-100 dataset (distilling WRN-40-2 to WRN-16-1), CSD achieved 60.88% accuracy, a significant jump from CMI's 56.46%. * Robustness to Heterogeneity: The framework proved effective even when teacher and student architectures were vastly different (e.g., Wide ResNet to MobileNet-V2), a scenario where traditional DFKD often fails. * Stability: By avoiding noisy teacher labels for adversarial guidance, CSD resulted in smoother and more stable training curves compared to traditional adversarial methods.

Conclusion and Future Work

CSD demonstrates that student-centric data synthesis is a powerful paradigm for DFKD. By using rotation as a semantic proxy, the model identifies and fills gaps in the student's knowledge more effectively than teacher-centric methods.

While highly effective for natural images, the rotation-based task may be less suitable for datasets where orientation is invariant (e.g., medical imaging). Future research may explore more diverse self-supervised tasks or the integration of diffusion models to further enhance the quality and diversity of customized synthetic data.

Summary updated

BibTeX
@inproceedings{luo2023customizing,
  author = {Shiya Luo and Defang Chen and Can Wang},
  title = {{Customizing Synthetic Data for Data-Free Student Learning}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00312},
}

Loading related content...

Figure