Abstract
Dataset condensation aims to synthesize datasets with a few representative samples that can effectively represent the original datasets. This enables efficient training and produces models with performance close to those trained on the original sets. Most existing dataset condensation methods conduct dataset learning under the bilevel (inner- and outer-loop) based optimization. However, the preceding methods perform with limited dataset generalization due to the notoriously complicated loss landscape and expensive time-space complexity of the inner-loop unrolling of bilevel optimization. These issues deteriorate when the datasets are learned via matching the trajectories of networks trained on the real and synthetic datasets with a long horizon inner-loop. To address these issues, we introduce Sharpness-Aware Trajectory Matching (SATM), which enhances the generalization capability of learned synthetic datasets by optimising the sharpness of the loss landscape and objective simultaneously. Moreover, our approach is coupled with an efficient hypergradient approximation that is mathematically well-supported and straightforward to implement along with controllable computational overhead. Empirical evaluations of SATM demonstrate its effectiveness across various applications, including in-domain benchmarks and out-of-domain settings.
Introduction
Dataset Condensation (DC) seeks to reduce massive datasets into a tiny set of synthetic images that retain the original's performance. A common approach is Trajectory Matching (MTT), where synthetic images are optimized so that a model trained on them follows the weight sequence ("trajectory") of a model trained on real data. However, DC often suffers from "brittleness," where synthetic datasets overfit to the specific neural network used during condensation, leading to poor generalization across different architectures.
We propose Sharpness-Aware Trajectory Matching (SATM) to address this. SATM improves the generalization of synthetic datasets by smoothing their optimization landscape. By jointly minimizing the distance between real and synthetic training trajectories while penalizing "sharp" loss regions, SATM creates condensed datasets that perform significantly better across different model architectures and out-of-domain tasks.
Methodology
SATM integrates Sharpness-Aware Minimization (SAM) principles into the bilevel optimization framework of dataset condensation.
Sharpness Optimization and Landscape Smoothing
Instead of solely minimizing the trajectory gap, SATM identifies synthetic images in "flat" regions of the loss landscape. This ensures that small perturbations in the synthetic data do not lead to significant performance drops. To stabilize optimization, a fluctuation strategy is employed where synthetic images are perturbed with Gaussian noise during the gradient ascent step.
Efficiency Strategies
To manage the high computational cost of trajectory matching, we propose several efficiency measures:
- Truncated Unrolling Hypergradient (TUH): Backpropagation is limited to the last few steps of the inner training loop, reducing memory usage.
- Trajectory Reusing (TR): The trajectory from the first gradient calculation is reused for the second "sharpness" calculation, avoiding a full second unrolling.
- Closed-form Learning Rate: A mathematically supported first-order approximation is used for learning the inner-loop learning rate, bypassing expensive second-order derivatives.
Experimental Evaluation
The method was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet-1K.
Key Results
- State-of-the-Art Performance: SATM consistently outperformed competitors like MTT, FTD, and TESLA. On CIFAR-10 (50 IPC), it achieved 73.9% accuracy.
- Cross-Architecture Superiority: When condensed on a ConvNet but tested on ResNet18, SATM maintained higher accuracy than MTT (e.g., 47.7% vs 46.4% at 10 IPC), demonstrating superior generalization.
- ImageNet-1K Success: SATM achieved 29.2% accuracy on ImageNet-1K (50 IPC), outperforming TESLA (27.9%) while maintaining competitive iteration speeds.
Discussion: Stability and Flatness
A core insight of this work is that the "outer-loop" optimization landscape (how synthetic images are updated) is naturally sharp, leading to overfitting. By forcing synthetic images into flatter regions, the data becomes more robust.
As shown in the stability analysis, SATM exhibits significantly lower variance and fewer "spikes" in hypergradient norms during training compared to standard MTT. This indicates a smoother, more stable learning process that directly contributes to better generalization.
Conclusion and Limitations
SATM provides a flexible, "plug-and-play" framework for dataset condensation that can be combined with other advanced optimizers. However, like other trajectory-matching methods, it still requires pre-calculated "expert" trajectories, which involves significant storage overhead. Future work may explore implicit gradients to manage long-horizon inner loops more effectively without storing intermediate weights.
Summary updated