This paper, presented at NeurIPS 2026, introduces a novel approach to dataset distillation that moves away from traditional classifier-based optimization in favor of direct distribution matching.
TL;DR
The authors propose "Drifting," a method that learns a tiny synthetic dataset by pushing and pulling samples in a latent feature space to match the real data distribution. By optimizing for distribution coverage rather than just classification accuracy, they produce high-quality, diverse synthetic samples that significantly outperform existing methods across multiple modalities, including images, audio, and video.
What Problem Does This Paper Address?
Dataset distillation aims to compress a massive dataset (like ImageNet) into a handful of synthetic "representative" images. Ideally, training a model on these few images should yield performance similar to training on the full dataset.
Historically, most methods have optimized these synthetic images to satisfy a specific classifier. However, this often leads to "homogeneous" data—images that look very similar because they only need to capture the features relevant to a specific decision boundary. These distilled sets often inherit the biases of the teacher model and fail to capture the rich diversity of the original data distribution, limiting their usefulness for other tasks or different model architectures.
Methods
The core innovation is Drifting for Dataset Distillation. Instead of using a classification loss, the authors treat the synthetic samples as particles in a force field within the latent space of a pre-trained feature extractor (like DINOv3 or MoCo-v3).
- Attractive and Repulsive Forces: Each synthetic sample is simultaneously pulled toward every real sample (attraction) and pushed away from every other synthetic sample (repulsion). This "drifting" process naturally encourages the synthetic data to spread out and cover the entire manifold of the real data.
- Augmentation Drifting: Recognizing that a few images cannot cover a whole distribution, the method drifts multiple augmentations of each synthetic image. This ensures that the "neighborhood" of each distilled image effectively covers a larger portion of the real data space. Visualizing how augmentations help cover the real distribution
- Laplacian Pyramid Parameterization: Rather than optimizing raw pixels, the images are represented as a Laplacian pyramid—a cascade of images at progressively higher resolutions. This acts as an implicit regularizer, preventing the optimization from creating the "noisy" adversarial patterns common in model inversion. The Laplacian pyramid reconstruction process
Datasets
The authors evaluate their method on several benchmarks:
- ImageNet-1k: The primary large-scale benchmark for image classification.
- ImageNette & ImageWoof: Smaller, specialized subsets of ImageNet used for rapid iteration.
- FMA (Free Music Archive): Used to demonstrate generalization to audio distillation.
- HMDB51: A human motion dataset used for video distillation.
Key Insights and Ideas
- Distribution Matching > Classification Objectives: By ignoring the classifier during distillation, the resulting images are more diverse and realistic. Surprisingly, this "task-agnostic" data actually performs better when used to train classifiers from scratch than data specifically optimized for classification.
- Self-Supervised Features are Superior: The choice of the latent space matters. Features from self-supervised models (DINOv3, MoCo-v3) produce much better distilled data than supervised models (ResNet-18). This suggests that self-supervised models capture a more "complete" representation of the data distribution.
- Model Interpretability: Distilling the same class through different backbones reveals the inherent biases of those models. For example, CLIP-distilled images appear "murkier," likely due to its text-alignment objective, whereas DINOv3 produces highly detailed, clear objects. Comparing how different models "see" the same class
Key Results
The quantitative improvements are substantial, particularly in the "from-scratch" training regime where synthetic data is most valuable.
- ImageNet-1k Performance: At 10 images per class (IPC), the MoCo-v3 variant achieves 27.0% hard-label accuracy and 40.3% soft-label accuracy on a ResNet-18. This is a massive leap over the previous noise-initialized state-of-the-art (G-VBSM), which only reached 11.7% and 29.8% respectively. Quantitative comparison against baselines
- Distribution Coverage: The authors propose a "Coverage" metric to measure how much of the real validation set is represented by the synthetic data. Their DINOv3 variant covers 77.0% of the distribution at 10 IPC, compared to less than 10% for most existing noise-initialized methods.
- Qualitative Quality: As seen in Figure 1, the distilled images are remarkably clear and diverse, looking like high-quality natural photos rather than abstract textures. High-quality distilled samples across various classes
Limitations and Future Work
The primary drawback is computational cost. Distilling 10 images per class for ImageNet-1k takes approximately 250 GPU hours. While the process is easily parallelizable (since each class is distilled independently), the total compute footprint is significant.
Future work could explore ways to speed up the drifting process or investigate how these highly diverse synthetic datasets might benefit other areas like generative model pre-training or privacy-preserving data sharing.
Summary updated