Abstract
Dataset distillation (DD) aims to condense large datasets into small synthetic sets while preserving model performance. Despite numerous proposed methods, a unified theoretical understanding of their scaling behavior and cross-configuration robustness has been lacking. This paper introduces the Configuration-Dynamics-Error (CDE) framework, establishing two fundamental laws: the $1/\sqrt{k}$ Scaling Law and the Configuration-Coverage Law. These laws define the "utility boundary" of distilled data, proving that synthetic dataset size must scale linearly with the diversity of target training configurations to maintain performance.
Introduction
Dataset Distillation (DD) is the process of condensing massive datasets (e.g., ImageNet) into a small set of synthetic "prototype" images. Ideally, a model trained on these few images achieves accuracy comparable to one trained on the full dataset, significantly reducing storage and compute requirements.
However, the field has lacked a unifying theory. Various "matching" strategies—Gradient Matching (GM), Distribution Matching (DM), and Trajectory Matching (TM)—have been treated as independent heuristics. Furthermore, distilled data often fails when the training "configuration" (e.g., architecture, optimizer, or augmentation) changes. This research addresses these gaps by defining the "utility boundary" of synthetic data through a unified theoretical lens.
The Configuration-Dynamics-Error (CDE) Framework
The authors propose the Configuration-Dynamics-Error (CDE) framework, which treats a training setup as a "configuration" that induces specific "optimization dynamics." The framework employs a bi-level perspective:
- Inner Loop: Models how parameters evolve when trained on distilled data.
- Outer Loop: Updates distilled data to minimize the "matching discrepancy" between synthetic and real optimization paths.
By reformulating DD through generalization error, the authors prove that GM, DM, and TM are interchangeable surrogates. They all aim to minimize the same underlying discrepancy using different mathematical shortcuts.
Fundamental Laws of Dataset Distillation
The framework yields two primary laws that govern the performance of distilled datasets:
1. The $1/\sqrt{k}$ Scaling Law
The performance gap between distilled and real data performance shrinks at a rate of $1/\sqrt{k}$, where $k$ is the number of distilled samples. Crucially, the law identifies an irreducible error floor. This explains the "IPC (Images Per Class) Saturation" effect: beyond a certain point, adding more synthetic images yields diminishing returns because the fundamental limit of the specific training configuration has been reached.
2. The Configuration-Coverage Law
This law dictates that for distilled data to remain effective across diverse models and optimizers (high "configuration diversity"), the size of the distilled dataset must grow linearly with that diversity. In essence, there is no "free lunch" for robustness; supporting a wider range of training settings requires more synthetic data.
Empirical Validation
The theoretical laws were validated across standard vision benchmarks, including MNIST, CIFAR-10/100, and ImageNette, using various architectures (ConvNet, ResNet-18, LeNet) and optimizers (SGD, Adam).
- Scaling Law Consistency: Across all datasets, the $1/\sqrt{k}$ scaling law held with high statistical significance ($R^2 > 0.85$).
- Robustness Trade-offs: Experiments confirmed the Coverage Law; as the variety of target architectures increased, generalization error rose unless the distilled sample size was increased proportionally.
- Method Efficiency: While the scaling law is universal, methods differ in "slope." Gradient Matching (GM) often achieves lower error within a specific configuration but degrades faster than Distribution Matching (DM) when configurations shift.
Limitations and Future Work
The current theory relies on "PL-type" (Polyak–Łojasiewicz) dynamics, which are common in smooth optimization but may not fully capture highly non-convex or adversarial scenarios. Additionally, the "Coverage Law" currently focuses on algorithmic diversity (optimizers, architectures) rather than semantic shifts (e.g., cross-domain transfer). Future research will aim to extend these laws to non-smooth optimization landscapes and data-level diversity.
Summary updated