Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws

Zhengquan Luo, Zhiqiang Xu

Abstract

Dataset distillation (DD) aims to condense large datasets into small synthetic sets while preserving model performance. Despite numerous proposed methods, a unified theoretical understanding of their scaling behavior and cross-configuration robustness has been lacking. This paper introduces the Configuration-Dynamics-Error (CDE) framework, establishing two fundamental laws: the $1/\sqrt{k}$ Scaling Law and the Configuration-Coverage Law. These laws define the "utility boundary" of distilled data, proving that synthetic dataset size must scale linearly with the diversity of target training configurations to maintain performance.

Introduction

Dataset Distillation (DD) is the process of condensing massive datasets (e.g., ImageNet) into a small set of synthetic "prototype" images. Ideally, a model trained on these few images achieves accuracy comparable to one trained on the full dataset, significantly reducing storage and compute requirements.

However, the field has lacked a unifying theory. Various "matching" strategies—Gradient Matching (GM), Distribution Matching (DM), and Trajectory Matching (TM)—have been treated as independent heuristics. Furthermore, distilled data often fails when the training "configuration" (e.g., architecture, optimizer, or augmentation) changes. This research addresses these gaps by defining the "utility boundary" of synthetic data through a unified theoretical lens.

The Configuration-Dynamics-Error (CDE) framework unifies dataset distillation methods and defines the scaling and coverage laws that govern synthetic data utility.

The Configuration-Dynamics-Error (CDE) Framework

The authors propose the Configuration-Dynamics-Error (CDE) framework, which treats a training setup as a "configuration" that induces specific "optimization dynamics." The framework employs a bi-level perspective:

  1. Inner Loop: Models how parameters evolve when trained on distilled data.
  2. Outer Loop: Updates distilled data to minimize the "matching discrepancy" between synthetic and real optimization paths.

By reformulating DD through generalization error, the authors prove that GM, DM, and TM are interchangeable surrogates. They all aim to minimize the same underlying discrepancy using different mathematical shortcuts.

Comparison of major dataset distillation objectives (DM, GM, TM) within the CDE framework, highlighting their mathematical relationships and trade-offs.

Fundamental Laws of Dataset Distillation

The framework yields two primary laws that govern the performance of distilled datasets:

1. The $1/\sqrt{k}$ Scaling Law

The performance gap between distilled and real data performance shrinks at a rate of $1/\sqrt{k}$, where $k$ is the number of distilled samples. Crucially, the law identifies an irreducible error floor. This explains the "IPC (Images Per Class) Saturation" effect: beyond a certain point, adding more synthetic images yields diminishing returns because the fundamental limit of the specific training configuration has been reached.

2. The Configuration-Coverage Law

This law dictates that for distilled data to remain effective across diverse models and optimizers (high "configuration diversity"), the size of the distilled dataset must grow linearly with that diversity. In essence, there is no "free lunch" for robustness; supporting a wider range of training settings requires more synthetic data.

Empirical Validation

The theoretical laws were validated across standard vision benchmarks, including MNIST, CIFAR-10/100, and ImageNette, using various architectures (ConvNet, ResNet-18, LeNet) and optimizers (SGD, Adam).

  • Scaling Law Consistency: Across all datasets, the $1/\sqrt{k}$ scaling law held with high statistical significance ($R^2 > 0.85$).
  • Robustness Trade-offs: Experiments confirmed the Coverage Law; as the variety of target architectures increased, generalization error rose unless the distilled sample size was increased proportionally.
  • Method Efficiency: While the scaling law is universal, methods differ in "slope." Gradient Matching (GM) often achieves lower error within a specific configuration but degrades faster than Distribution Matching (DM) when configurations shift.

Limitations and Future Work

The current theory relies on "PL-type" (Polyak–Łojasiewicz) dynamics, which are common in smooth optimization but may not fully capture highly non-convex or adversarial scenarios. Additionally, the "Coverage Law" currently focuses on algorithmic diversity (optimizers, architectures) rather than semantic shifts (e.g., cross-domain transfer). Future research will aim to extend these laws to non-smooth optimization landscapes and data-level diversity.

Summary updated

Abstract

Dataset distillation (DD) aims to construct compact synthetic datasets that allow models to achieve comparable performance to full-data training while substantially reducing storage and computation. Despite rapid empirical progress, its theoretical foundations remain limited: existing methods (gradient, distribution, trajectory matching) are built on heterogeneous surrogate objectives and optimization assumptions, which makes it difficult to analyze their common principles or provide general guarantees. Moreover, it is still unclear under what conditions distilled data can retain the effectiveness of full datasets when the training configuration, such as optimizer, architecture, or augmentation, changes. To answer these questions, we propose a unified theoretical framework, termed configuration–dynamics–error analysis, which reformulates major DD approaches under a common generalization-error perspective and provides two main results: (i) a scaling law that provides a single-configuration upper bound, characterizing how the error decreases as the distilled sample size increases and explaining the commonly observed performance saturation effect; and (ii) a coverage law showing that the required distilled sample size scales linearly with configuration diversity, with provably matching upper and lower bounds. In addition, our unified analysis reveals that various matching methods are interchangeable surrogates, reducing the same generalization error, clarifying why they can all achieve dataset distillation and providing guidance on how surrogate choices affect sample efficiency and robustness. Experiments across diverse methods and configurations empirically confirm the derived laws, advancing a theoretical foundation for DD and enabling theory-driven design of compact, configuration-robust dataset distillation.
BibTeX
@misc{luo2025utility,
  author = {Zhengquan Luo and Zhiqiang Xu},
  title = {{Utility Boundary of Dataset Distillation: Scaling and Configuration-Coverage Laws}},
  year = {2025},
  eprint = {2512.05817},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.05817},
}
Figure