Flexible and Scalable Dataset Distillation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 218 papers22 conceptsfrozen 2026-10-04

Dataset distillation aims to condense massive datasets into a handful of synthetic samples that retain the original's predictive power. Traditionally, this process is rigid: you choose a budget (e.g., 10 images per class), run a costly optimization, and receive a static set. If your deployment constraints change—perhaps a mobile device has less memory than expected—you must restart the entire distillation process from scratch.

Recent research has shifted toward flexible distillation, creating synthetic representations that can be dynamically resized or sampled to fit varying resource constraints without retraining.

Moving from Static Sets to Generative Models

The most significant shift in building intuition for this field is moving away from thinking of distilled data as a "folder of images." Instead, researchers are distilling datasets into generative models.

In DiM: Distilling Dataset into Generative Model, Kai Wang and colleagues propose training a generator to capture the dataset's essence. This allows for "on-the-fly" generation of training samples. If you need more diversity, you simply sample more from the model. Similarly, Longzhen Li and Guang Li demonstrate in Generative Dataset Distillation that optimizing a GAN to balance global semantic structure with local textures produces data that generalizes better across different architectures than static pixels do.

To solve the speed bottleneck of training these generators, Songhua Liu and Xinchao Wang introduced MGDD, which uses meta-learning to adapt a pre-trained generator to new target datasets rapidly, achieving massive speedups in redeployment.

The "Condense Once, Use Anywhere" Philosophy

If you prefer discrete images over generative models, the challenge is making a large distilled set "prunable." Normally, if you take a distilled set of 50 images and randomly pick 10, performance collapses because the 50 were optimized to work as a collective unit.

Yang He and Joey Tianyi Zhou addressed this in You Only Condense Once (YOCO). They developed rules for pruning condensed datasets by prioritizing "easy" samples and maintaining class balance, allowing a single high-density set to be resized for any device. They expanded this in Multisize Dataset Condensation (MDC) by introducing an adaptive subset loss. This ensures that any subset of the distilled data—not just the full set—remains a valid representation of the original distribution.

Adaptive Complexity in Inference and Modalities

This drive for flexibility extends beyond training data into model architecture and multimodal analysis:

  • Inference Flexibility: In Clockwork Diffusion, Amir Ghodrati and Amirhossein Habibian show that distillation can be used to make diffusion models flexible. By distilling heavy layers into lightweight adaptors, they allow the model to skip computations dynamically based on the required image quality.
  • Input Flexibility: Haosen Yang and Josien P. W. Pluim apply these concepts to medical AI with Adaptive Prototype Learning. Instead of using a fixed number of summary points for every patient, their model uses learnable queries to distill varying amounts of information depending on the complexity of the case.

By decoupling the distillation process from the final deployment size, these researchers have transformed dataset distillation from a rigid optimization task into a versatile tool for resource-constrained machine learning.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 10 papers
  2. Version 1 · 2026-10-04 · 18 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_size_synthetic_data_constraint_1c404097,
  title = {fixed-size synthetic data constraint},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-size-synthetic-data-constraint/rdr_1c404097f62e4647867104c4d8bcc948},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_flexible_and_scalable_dataset_distillati_1c404097_v1,
  title = {Flexible and Scalable Dataset Distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/flexible-and-scalable-dataset-distillation/rdr_1c404097f62e4647867104c4d8bcc948/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}