Abstract
Training a large and state-of-the-art machine learning model typically necessitates the use of large-scale datasets, which, in turn, makes the training and parameter-tuning process expensive and time-consuming. Some researchers opt to distil information from real-world datasets into tiny and compact synthetic datasets while maintaining their ability to train a well-performing model, hence proposing a data-efficient method known as Dataset Distillation (DD). Despite recent progress in this field, existing methods still underperform and cannot effectively replace large datasets. In this paper, unlike previous methods that focus solely on improving the efficacy of student distillation, we are the first to recognize the important interplay between expert and student. We argue the significant impact of expert smoothness when employing more potent expert trajectories in subsequent dataset distillation. Based on this, we introduce the integration of clipping loss and gradient penalty to regulate the rate of parameter changes in expert trajectories. Furthermore, in response to the sensitivity exhibited towards randomly initialized variables during distillation, we propose representative initialization for synthetic dataset and balanced inner-loop loss. Finally, we present two enhancement strategies, namely intermediate matching loss and weight perturbation, to mitigate the potential occurrence of cumulative errors. We conduct extensive experiments on datasets of different scales, sizes, and resolutions. The results demonstrate that the proposed method significantly outperforms prior methods.
Introduction
Dataset Distillation (DD) aims to condense massive datasets into a handful of highly informative synthetic images. A successful distillation allows a model trained on this tiny subset to achieve performance comparable to one trained on the full dataset. A leading paradigm in this field is Trajectory Matching (MTT), which trains a "student" model on synthetic data to mimic the parameter path (trajectory) of an "expert" model trained on real data.
However, existing MTT methods face a "Catastrophic Discrepancy." When high-performance optimizers like Adam or SGD with Momentum are used to train the expert, distillation performance often collapses. This occurs because these experts move through parameter space too erratically for the student to follow. Without the expert's historical gradient momentum, the student diverges, resulting in "black hole" synthetic images that resemble random noise.
The AST Framework
To address these challenges, we propose AST (Alignment with Smooth and high-quality expert Trajectories). AST focuses on making the expert a better "teacher" by ensuring its trajectories are smooth, stable, and accessible to the student model.
1. Smooth Expert Trajectories
To utilize powerful optimizers without the associated jitter, AST introduces Clipping Loss and a Gradient Penalty. These constraints enforce Lipschitz continuity on the expert model, ensuring that parameter changes are gradual and predictable. This smoothness is critical for the student model to maintain alignment throughout the distillation process.
2. Balancing Stochasticity
Distillation is highly sensitive to initial conditions and training dynamics. AST mitigates this through two techniques:
- Representative Initialization: Instead of random noise, we use K-Means clustering on expert feature maps to select the most central real images as the starting point for synthetic data.
- Balanced Inner-loop Loss: We apply a weighting coefficient ($\nu$) to the distillation loss. This prevents early, high-loss epochs from disproportionately influencing the synthetic data, leading to more stable convergence.
3. Error Mitigation
To prevent small alignment errors from snowballing into training failure, AST employs Intermediate Matching Loss—checking alignment multiple times within a single trajectory segment—and Weight Perturbation. By adding slight noise to expert weights during the outer-loop, we improve the robustness of the student's learning path.
Experimental Evaluation
AST was evaluated across several standard benchmarks, including CIFAR-10/100, Tiny ImageNet (64x64), and high-resolution ImageNet subsets (128x128) such as ImageNette and ImageWoof.
Key results include:
- State-of-the-Art Performance: On CIFAR-100 with 10 images per class, AST achieved 44.4% accuracy, surpassing previous benchmarks. On Tiny ImageNet, it reached 25.7%, a significant improvement over the prior best of 24.5%.
- Efficiency: AST demonstrates remarkable convergence speed. On CIFAR-100, it reaches the final performance of previous methods in just 500 iterations, compared to the 5,000 required by MTT.
- Generalization: Synthetic datasets created with AST generalize well across architectures, with ConvNet-distilled data performing strongly when transferred to ResNet or VGG models.
Conclusion
AST shifts the focus of dataset distillation from the student's learning capacity to the expert's "teachability." By enforcing smoothness in expert trajectories and mitigating stochastic errors, AST enables the use of more powerful optimizers and higher-quality experts. This approach not only sets new accuracy records but also significantly reduces the computational cost of the distillation process, moving us closer to efficient, large-scale data condensation.
Summary updated