In the quest to train smaller, faster models, researchers have long relied on Knowledge Distillation (KD)—the process where a complex "teacher" model guides a lightweight "student." The secret sauce of this process is the "soft label": instead of telling a student that an image is simply a "dog," the teacher provides a full probability distribution (e.g., 80% dog, 15% cat, 5% car). This "dark knowledge" captures the rich relationships between classes that a simple hard label misses.
However, as we scale to massive datasets like ImageNet-1K or trillion-token language corpora, a "soft label trap" emerges. Storing these high-dimensional probability distributions for every training sample creates a massive storage bottleneck that can exceed the size of the original raw data by orders of magnitude. Recent research focuses on breaking this bottleneck by rethinking how we represent and store teacher knowledge.
The Soft Label Trap
The primary goal of Dataset Distillation (DD) is to condense massive datasets into tiny, synthetic versions that retain the original's performance. As Youneng Bao and Yiping Liu observe, the high-precision soft labels required for these synthetic images often consume more storage space than the images themselves.
This creates a paradox: a "compressed" dataset might actually require more disk space to store its metadata than the original uncompressed dataset. Lingao Xiao and Yang He highlight that in large-scale distillation, auxiliary soft labels can be 30x larger than the synthetic images, reaching terabytes for datasets with thousands of classes. This overhead undermines the portability and efficiency that distillation is intended to provide.
Moving from Dynamic to Static Knowledge
One of the most effective ways to reduce this footprint is to change when and how we generate labels. Traditional distillation often uses "dynamic labels" generated by the teacher in every training epoch. While accurate, this requires the teacher model to be held in memory alongside the student, creating a hardware bottleneck Shicheng Tan.
To solve this, Xin Zhang and Joey Tianyi Zhou introduced INFER, which utilizes "static soft labels." By averaging teacher model outputs into a single fixed distribution per instance, they reduced the storage footprint of distilled datasets by 99.3%. While there is a slight trade-off in peak accuracy compared to dynamic labels, the efficiency gain makes large-scale distillation practical for resource-constrained environments.
Hard Labels as Semantic Anchors
If soft labels are too heavy and hard labels are too simplistic, can we use both? Jiacheng Cui and Zhiqiang Shen proposed HALD, a paradigm that strategically reintroduces hard labels as "content-agnostic anchors."
In dataset distillation, synthetic images are often subjected to heavy augmentations (like random crops). Soft labels can "drift" when a crop captures a non-representative part of an image. By alternating between soft labels and hard labels, HALD uses the hard label to calibrate this semantic drift. This approach not only boosted accuracy on ImageNet-1K by 9% but also achieved a 100x reduction in storage by minimizing the need for high-precision teacher metadata.
Sparsification and Importance Sampling
In the realm of Large Language Models (LLMs), the storage problem is even more acute. Storing full probability distributions for a 50,000-word vocabulary across trillions of tokens would require petabytes of storage Hao Peng.
Researchers are now applying sparsification techniques to teacher outputs:
- Top-K Truncation: Storing only the most likely predictions. While common, this can introduce bias.
- Importance Sampling: Anshumann developed RS-KD, which uses random sampling to train students using only 0.01% of the teacher's output data. This reduced storage requirements by 25x while maintaining performance nearly identical to full distillation.
- Token Labeling: Zihang Jiang and Xiaojie Jin shifted the focus from global labels to dense, patch-level supervision, using "machine annotators" to provide rich signals without the overhead of full-image distributions.
Distillation as a Rate-Utility Problem
The field is moving toward a formal information-theoretic view of distillation. Youneng Bao reframes dataset distillation as a Rate-Utility optimization problem. Instead of just trying to match the teacher's accuracy, researchers are now jointly optimizing for the "rate" (the total bits required to store images and labels) and the "utility" (the final model performance).
This perspective has led to innovations like distance-based label compression Minh-Tuan Tran and the PCA (Prune, Combine, and Augment) framework Lingao Xiao, which achieves state-of-the-art compression by abandoning soft labels entirely in favor of high-quality real image subsets and hard labels.
By treating the teacher's knowledge not as a fixed requirement but as a signal to be compressed, these researchers are making it possible to distill the world's largest datasets into portable, efficient packages.