Dataset distillation aims to compress a massive dataset into a tiny set of synthetic "representative" samples. When a model trains on this small synthetic set, it should achieve performance nearly identical to training on the full original data. Traditionally, these synthetic samples start as random noise and are iteratively optimized to match the gradients or distributions of the real data.
However, starting from pure noise is computationally expensive and often leads to "optimization collapse," where the synthetic samples fail to capture the diversity of the original distribution. Recent research has shifted toward structured initialization, where synthetic data begins not as noise, but as carefully selected "warm starts" derived from the source data.
The Cold Start Problem in Condensation
In standard dataset condensation, the optimization process must simultaneously learn the structural features of a class (e.g., what a "dog" looks like) and the specific informative features that help a model converge quickly. When initialized with random noise, the optimizer spends significant budget simply trying to reach a recognizable image manifold.
Tom De Schepper and Kevin Mets highlight in Dataset Condensation with Latent Quantile Matching that the initial state of synthetic samples is the primary bottleneck for distribution matching. If the starting point is too far from the real data distribution, the optimization may trap the synthetic samples in poor local minima that lack representative diversity.
Clustering as a Semantic Anchor
One of the most effective ways to provide a "warm start" is to initialize synthetic samples using the centers of data clusters. By performing K-means or similar clustering on the original feature space, researchers can identify the "prototypes" of a dataset.
In the DREAM framework, Yanqing Liu and colleagues demonstrate that initializing with sub-cluster centers places synthetic data in high-density regions of the feature space from the outset. This strategy achieved an 8x speedup in training because the optimizer only needs to "refine" these prototypes rather than "invent" them. Similarly, Ding Qi and Jian Li found that for continuous, non-categorical data, cluster centroids ensure the synthetic set covers the entire semantic distribution, preventing the redundancy that often plagues random sampling.
This principle extends beyond images. For time series data, Zhanyu Liu showed in CondTSC that starting with real-data centroids is critical for preserving high-frequency temporal dependencies that are nearly impossible to recover from noise.
Coresets and Provable Subsets
While clustering finds the "average" of a group, another approach looks for the most "informative" individual samples. This is often done through coreset selection—finding a mathematical subset of data that approximates the loss of the entire set.
Murad Tukan and Alaa Maalouf argue in Dataset Distillation Meets Provable Subset Selection that using coreset theory to initialize synthetic sets provides a provably robust starting point. By selecting points that capture the underlying geometry of the distribution, the optimization process starts much closer to a global minimum. Xueluan Gong applied a similar logic to model extraction attacks in InverseNet, using coresets to ensure a substitute model captures maximum variance with a minimal query budget.
Selecting for Difficulty and Structure
Not all real samples are equally good starting points. Some researchers suggest that the "hardness" of a sample determines its value as a seed for distillation.
- High-Loss Samples: In MKDT, Baharan Mirzasoleiman and Jiayi Ni initialize synthetic images using samples that expert models find most challenging. These "hard" samples contain the most informative features for self-supervised pre-training.
- Medium-Difficulty Patches: Conversely, Ammar Sherif and Zhiqiang Shen introduce DELT, which uses medium-difficulty patches. They argue these provide enough structure to accelerate convergence while remaining flexible enough for the gradient-based refinement to add new information.
- Structural Foundations: For high-resolution tasks like image super-resolution, Brian Bernhard Moser found that initializing with downscaled versions of real images is essential. This provides a structural foundation that allows the model to focus on learning intricate textures rather than basic shapes.
Why It Works: Flattening the Landscape
The shared intuition across these papers is that the optimization landscape of dataset distillation is notoriously "jagged." Random initialization often leaves the synthetic samples in a region where gradients are chaotic or vanishing.
By using structured initialization—whether through centroids (Wang et al.), real samples (Shao et al.), or mastery profiles (Rebecca Nugent)—researchers are effectively placing the synthetic data on a "downhill" path toward a high-quality representation. This shift transforms dataset distillation from a task of creation into a task of refinement, making the process faster, more stable, and significantly more accurate.