What Makes a Good Dataset for Knowledge Distillation?

Logan Frank, Jim DavisCVPR 2025 · DOI

Abstract

Knowledge Distillation (KD) is a powerful technique for model compression, typically requiring access to the original training data used for the teacher model. However, in many real-world scenarios, this data is proprietary or unavailable. This paper investigates the fundamental requirements for surrogate datasets in KD. We demonstrate that visual realism is secondary to the ability of a dataset to thoroughly sample a teacher's decision boundaries. Remarkably, we find that procedurally generated, "unnatural" imagery—such as OpenGL shaders—can serve as highly effective surrogate data, often matching the performance of original in-domain datasets.

Introduction

Knowledge Distillation (KD) involves training a compact "student" network to mimic the output distribution of a larger, pre-trained "teacher" network. While highly effective, standard KD assumes the availability of the teacher's original training set. When this data is missing, practitioners often turn to surrogate datasets. This research asks: What properties make a surrogate dataset effective for distillation? Does the data need to be "in-domain," or can we leverage synthetic, non-natural data to extract the teacher's knowledge?

Methodology

The study systematically evaluates various surrogate data sources, ranging from real-world images to purely synthetic patterns. The researchers explored four primary strategies:

  1. Surrogate Diversity: Comparing real in-domain (ID) data, real out-of-domain (OOD) data (e.g., using ImageNet to distill a CIFAR-10 teacher), and synthetic data.
  2. Procedural Generation: The authors utilized OpenGL shaders and simple geometric "Leaves" to generate infinite streams of data. These images lack semantic meaning but possess rich textures and complex geometric features.
  3. Intensive Augmentation: Because synthetic data has no fixed semantic labels, the authors applied aggressive data augmentation (including RandAugment and Mixup) to maximize the coverage of the input space.
  4. Adversarial Refinement: To improve "weak" surrogate sets, the authors introduced minor adversarial perturbations. These tweaks force the teacher to reveal its decision boundaries more clearly, allowing the student to learn the teacher's underlying logic more efficiently.

Key Insights: KD as a Sampling Problem

The authors propose that distillation is essentially a signal reconstruction task. To successfully replicate the teacher's behavior, the surrogate data must "sample" the teacher's output space evenly.

Visualizing decision boundaries for Teacher and Student models across different datasets.

As shown in the decision boundary visualizations, the student model is capable of learning complex boundary shapes even when trained on synthetic OpenGL data. The effectiveness of a dataset is tied to its relative entropy: a good dataset produces a diverse range of predictions from the teacher, rather than clustering around a single class. This explains why complex textures (OpenGL) outperform simple shapes (Leaves), which in turn outperform unstructured noise.

Empirical Results

The experimental results challenge the assumption that surrogate data must look like the original training set.

Performance comparison of student models distilled using various real and synthetic surrogate datasets.

Key findings include:

  • Synthetic Success: For a CIFAR-10 teacher, distilling with OpenGL shaders achieved an accuracy of 94.02%, remarkably close to the 95.98% achieved with the original data. On the EuroSAT (satellite) benchmark, the gap was even smaller (0.2%).
  • Diversity Over Domain: Large, diverse out-of-domain datasets (like ImageNet-OOD) frequently outperformed smaller in-domain datasets. The sheer volume and variety of features in OOD data provide a more comprehensive map of the teacher's decision space.
  • Adversarial Boosting: Adversarial perturbations proved highly effective at "fixing" poor domain alignment. For example, using aircraft images to distill a satellite imagery teacher normally yields poor results, but adversarial tweaks improved student accuracy by 38%.

Conclusion

This research demonstrates that the "realism" of a dataset is overrated for the purposes of knowledge distillation. Instead, the value of a surrogate dataset lies in its structural diversity and its ability to trigger a wide range of responses from the teacher model. While fine-grained tasks still present a challenge for synthetic data, the success of procedural shaders suggests a new path for data-free or data-constrained model compression, where infinite, privacy-preserving synthetic data can replace proprietary training sets.

Summary updated

BibTeX
@inproceedings{frank2025what,
  author = {Logan Frank and Jim Davis},
  title = {{What Makes a Good Dataset for Knowledge Distillation?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02212},
}
Figure