Dataset Distillation via Drifting

George Cazenavette, Angelina Quan, Giannis Daras, Antonio Torralba, Vincent SitzmannarXiv 2026 · arXiv

This paper, presented at NeurIPS 2026, introduces a novel approach to dataset distillation that moves away from traditional classifier-based optimization in favor of direct distribution matching.

TL;DR

The authors propose "Drifting," a method that learns a tiny synthetic dataset by pushing and pulling samples in a latent feature space to match the real data distribution. By optimizing for distribution coverage rather than just classification accuracy, they produce high-quality, diverse synthetic samples that significantly outperform existing methods across multiple modalities, including images, audio, and video.

What Problem Does This Paper Address?

Dataset distillation aims to compress a massive dataset (like ImageNet) into a handful of synthetic "representative" images. Ideally, training a model on these few images should yield performance similar to training on the full dataset.

Historically, most methods have optimized these synthetic images to satisfy a specific classifier. However, this often leads to "homogeneous" data—images that look very similar because they only need to capture the features relevant to a specific decision boundary. These distilled sets often inherit the biases of the teacher model and fail to capture the rich diversity of the original data distribution, limiting their usefulness for other tasks or different model architectures.

Methods

The core innovation is Drifting for Dataset Distillation. Instead of using a classification loss, the authors treat the synthetic samples as particles in a force field within the latent space of a pre-trained feature extractor (like DINOv3 or MoCo-v3).

  1. Attractive and Repulsive Forces: Each synthetic sample is simultaneously pulled toward every real sample (attraction) and pushed away from every other synthetic sample (repulsion). This "drifting" process naturally encourages the synthetic data to spread out and cover the entire manifold of the real data.
  2. Augmentation Drifting: Recognizing that a few images cannot cover a whole distribution, the method drifts multiple augmentations of each synthetic image. This ensures that the "neighborhood" of each distilled image effectively covers a larger portion of the real data space. Visualizing how augmentations help cover the real distribution
  3. Laplacian Pyramid Parameterization: Rather than optimizing raw pixels, the images are represented as a Laplacian pyramid—a cascade of images at progressively higher resolutions. This acts as an implicit regularizer, preventing the optimization from creating the "noisy" adversarial patterns common in model inversion. The Laplacian pyramid reconstruction process
Visualizing how augmentations help cover the real distribution
Figure 2: Augmentations of Distilled Images Cover the Real Distribution. At each distillation step, we take multiple augmentations of each synthetic sample (left). The synthetic data, together with their augmentations, approximately cover the distribution of real data (right). In this plot, the stars represent the distilled images, the triangles 10 augmentations of each distilled image, and the circles real data. Drifting multiple augmentations of each synthetic sample at each step leads to high intra-image diversity, allowing some distilled images (such as the orange star) to cover many parts of the real distribution at the same time.

Datasets

The authors evaluate their method on several benchmarks:

  • ImageNet-1k: The primary large-scale benchmark for image classification.
  • ImageNette & ImageWoof: Smaller, specialized subsets of ImageNet used for rapid iteration.
  • FMA (Free Music Archive): Used to demonstrate generalization to audio distillation.
  • HMDB51: A human motion dataset used for video distillation.

Key Insights and Ideas

  • Distribution Matching > Classification Objectives: By ignoring the classifier during distillation, the resulting images are more diverse and realistic. Surprisingly, this "task-agnostic" data actually performs better when used to train classifiers from scratch than data specifically optimized for classification.
  • Self-Supervised Features are Superior: The choice of the latent space matters. Features from self-supervised models (DINOv3, MoCo-v3) produce much better distilled data than supervised models (ResNet-18). This suggests that self-supervised models capture a more "complete" representation of the data distribution.
  • Model Interpretability: Distilling the same class through different backbones reveals the inherent biases of those models. For example, CLIP-distilled images appear "murkier," likely due to its text-alignment objective, whereas DINOv3 produces highly detailed, clear objects. Comparing how different models "see" the same class
Comparing how different models "see" the same class

Key Results

The quantitative improvements are substantial, particularly in the "from-scratch" training regime where synthetic data is most valuable.

  • ImageNet-1k Performance: At 10 images per class (IPC), the MoCo-v3 variant achieves 27.0% hard-label accuracy and 40.3% soft-label accuracy on a ResNet-18. This is a massive leap over the previous noise-initialized state-of-the-art (G-VBSM), which only reached 11.7% and 29.8% respectively. Quantitative comparison against baselines
  • Distribution Coverage: The authors propose a "Coverage" metric to measure how much of the real validation set is represented by the synthetic data. Their DINOv3 variant covers 77.0% of the distribution at 10 IPC, compared to less than 10% for most existing noise-initialized methods.
  • Qualitative Quality: As seen in Figure 1, the distilled images are remarkably clear and diverse, looking like high-quality natural photos rather than abstract textures. High-quality distilled samples across various classes
Quantitative comparison against baselines
Figure 1: A selection of 6 classes from ImageNet-1k [12] distilled to 10 images per class via drifting with DINOv3 [53] features. Images seen here were initialized with random noise. Our method yields synthetic data that are high in both quality and diversity by virtue of the attractive and repulsive forces of our drifting loss.

Limitations and Future Work

The primary drawback is computational cost. Distilling 10 images per class for ImageNet-1k takes approximately 250 GPU hours. While the process is easily parallelizable (since each class is distilled independently), the total compute footprint is significant.

Future work could explore ways to speed up the drifting process or investigate how these highly diverse synthetic datasets might benefit other areas like generative model pre-training or privacy-preserving data sharing.

Summary updated

Abstract

The task of dataset distillation concerns the learning of a small set of synthetic samples that can serve as an effective replacement for a large real dataset when used for downstream tasks. While many solutions have been proposed since the problem's inception, most of them primarily seek to optimize the distilled data with respect to a classifier. In this work, we propose a new distillation method that learns a set of samples such that they directly approximate the per-class data distribution. Specifically, we "drift" our distilled data in the latent space of a feature extractor such that they move both toward the real data and away from each other without using a classification objective. We find that our drifted data cover the real distribution significantly better than existing methods while still serving as powerful training samples for classification. Furthermore, we showcase our method's use as a visualization tool for better interpreting models and extend it beyond images to other modalities. Our work advances the frontier of dataset distillation, offering a novel synthesis method and proposing distribution-based metrics as a complementary evaluation. More broadly, our findings help address the growing need to understand both data and models by uncovering more about how they interact.
BibTeX
@misc{cazenavette2026dataset,
  author = {George Cazenavette and Angelina Quan and Giannis Daras and Antonio Torralba and Vincent Sitzmann},
  title = {{Dataset Distillation via Drifting}},
  year = {2026},
  eprint = {2611.16642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16642},
}
Figure