Enhancing Generalization via Sharpness-Aware Trajectory Matching for Dataset Condensation

Boyan Gao, Bo Zhao, Shreyank N Gowda, Xingrun Xing, Yibo Yang, Timothy Hospedales, David A. CliftonSubmission to ICLR 2026 · Rejected

TL;DR

Dataset condensation distills massive training sets into a handful of synthetic images, but current trajectory-matching methods produce datasets that overfit to specific architectures and sharp loss landscapes. Sharpness-Aware Trajectory Matching (SATM) addresses this by integrating sharpness-aware minimization into bilevel optimization alongside two computational shortcuts—truncated backpropagation and trajectory reuse—allowing condensed datasets to generalize significantly better across diverse architectures and tasks without ballooning compute budgets.

What Problem Does This Paper Address?

Dataset Condensation (DC) aims to synthesize small, information-dense image sets so that neural networks trained on them match the performance of models trained on full-scale datasets. A leading approach, Matching Training Trajectories (MTT), aligns model parameter trajectories trained on synthetic data with "expert" trajectories trained on real data.

However, MTT relies on deep bilevel optimization (unrolling the inner training loop over dozens of steps). This leads to two critical bottlenecks:

  1. Poor Generalization: The outer-loop optimization landscape is notoriously sharp and irregular, causing synthesized images to overfit to the exact model architecture and training dynamics used during distillation.
  2. Extreme Computational Overhead: Standard Sharpness-Aware Minimization (SAM)—which pushes parameters into flatter, more generalizable loss minima—requires two forward/backward passes per step. Naively applying SAM to long unrolled bilevel trajectories would double an already prohibitive memory and runtime cost.

Methods

SATM adapts sharpness-aware minimization to the outer-loop objective of trajectory matching while introducing efficiency-oriented approximations:

  • Landscape Smoothing via Fluctuation: Estimating the worst-case perturbation vector ($\epsilon$) using a single-step gradient ascent often fails in sharp, high-variance bilevel landscapes. SATM adds input-dependent Gaussian noise proportional to image norms ($\phi_j^\Delta \sim \mathcal{N}(0, \gamma \|\phi_j\|_2)$) before gradient ascent, smoothing the local landscape.
  • Truncated Unrolling Hypergradient (TUH): Instead of backpropagating through all $N$ inner-loop unrolling steps, SATM differentiates only through the final $N - \iota$ steps. The authors prove theoretically (Theorem 4.1) that the approximation error decays exponentially as the model approaches an inner-loop optimum.
  • Trajectory Reusing (TR): Rather than re-running the entire inner loop from step $0$ to evaluate the perturbed dataset $\phi + \epsilon$, SATM branches off from step $\tau$ of the unperturbed trajectory ($\hat{\theta}_\tau(\phi+\epsilon) \approx \theta_\tau(\phi)$). The authors bound the drift between these trajectories using a growth recursion bound (Theorem 4.2).
  • First-Order Inner Learning Rate Adaptation: Instead of expensive second-order hypergradient differentiation to tune the inner-loop learning rate $\alpha$, SATM derives a closed-form first-order approximation that aggregates step-wise gradients.
Trajectory 1 (ϕ):        θ_0 ──> ... ──> θ_τ ──> ... ──> θ_N (Evaluate ∇F(ϕ))
                                          │
Trajectory 2 (ϕ + ϵ):                     └───> ... ──> θ_N (Evaluate ∇F(ϕ + ϵ))
                               (Reused prefix; truncated unroll)

Datasets

The authors evaluate SATM on standard image classification and distillation benchmarks:

  • CIFAR-10 & CIFAR-100 ($32 \times 32$)
  • TinyImageNet (200 classes, $64 \times 64$)
  • ImageNet-1K ($224 \times 224$ full dataset)
  • High-Resolution Subsets: ImageNette, ImageWoof, ImageFruit, ImageMeow ($128 \times 128$)

No new datasets were introduced; the paper uses existing benchmarks and trajectory-matching evaluation protocols.

Key Insights and Ideas

  • Flatness Translates to Cross-Architecture Transferability: While existing methods like FTD match in-domain performance on the base training architecture, their synthesized images degrade sharply on unseen architectures. Flattening the outer-loop landscape directly counteracts this architecture-specific overfitting.
  • Early Unrolled Steps Provide Negligible Outer Gradients: Mathematical analysis shows that inner-loop memory chains can be aggressively pruned (retaining only the last 20–33% of steps) without meaningful loss in distilled data quality, eliminating the primary memory bottleneck of trajectory matching.
  • Trajectory Reuse Beats Re-computation: The trajectory drift induced by a small perturbation $\epsilon$ within the SAM ball ($\rho$) accumulates slowly enough that warm-starting the second inner loop midway through the trajectory introduces negligible error while halving the forward-pass burden.

Key Results

  • Standard Benchmarks: SATM consistently outperforms previous state-of-the-art methods (MTT, FTD, TESLA, MDC):
  • CIFAR-10: 49.0% (1 IPC), 67.1% (10 IPC), and 73.9% (50 IPC).
  • CIFAR-100: 26.1% (1 IPC), 45.2% (10 IPC), and 53.2% (50 IPC) — outperforming MTT's 47.7% at 50 IPC by +5.5%.
  • TinyImageNet: Achieves 29.4% at 50 IPC (vs. MTT's 28.0% and FTD's 28.2%).
  • ImageNet-1K Scalability: Where most trajectory-matching methods run out of memory, SATM reaches 30.2% at 50 IPC (surpassing TESLA’s 27.9%) with lower per-iteration runtime (44.7s vs. 46.8s).
  • Cross-Architecture Generalization: On CIFAR-10 (50 IPC), a ConvNet-synthesized dataset evaluated on unseen architectures shows large margins over FTD: AlexNet (56.9% vs. 53.8%), VGG11 (63.5% vs. 58.4%), and Vision Transformer / ViT (52.7% vs. 50.1%).
  • Continual Learning: Under task-incremental settings (e.g., 20-step incremental on TinyImageNet), SATM synthetic replay samples mitigate catastrophic forgetting better than DM, MTT, and FTD across all stages.

Limitations and Future Work

  • Residual Memory Demands: While memory is significantly reduced compared to MTT (e.g., drops from 80.5 GB to 26.5 GB on ImageNet-1K), SATM still consumes more memory than pure constant-memory algorithms like TESLA (17.4 GB) because it maintains truncated computation graphs.
  • Heuristic Alignment of Truncation Bounds: For simplicity, the authors set truncation step $\iota$ equal to reuse point $\tau$; tuning these separately could yield better efficiency-accuracy trade-offs.
  • Second-Order Free Alternatives: The authors identify implicit differentiation (avoiding unrolling computational graphs altogether) as an open direction to eliminate the second trajectory unrolling entirely.

Summary updated

Abstract

Dataset condensation aims to synthesize datasets with a few representative samples that can effectively represent the original datasets. This enables efficient training and produces models with performance close to those trained on the original sets. Most existing dataset condensation methods conduct dataset learning under the bilevel (inner- and outer-loop) based optimization. However, the preceding methods perform with limited dataset generalization due to the notoriously complicated loss landscape and expensive time-space complexity of the inner-loop unrolling of bilevel optimization. These issues deteriorate when the datasets are learned via matching the trajectories of networks trained on the real and synthetic datasets with a long horizon inner-loop. To address these issues, we introduce Sharpness-Aware Trajectory Matching (SATM), which enhances the generalization capability of learned synthetic datasets by optimizing the sharpness of the loss landscape and objective simultaneously. Moreover, our approach is coupled with an efficient hypergradient approximation that is mathematically well-supported and straightforward to implement, along with controllable computational overhead. Empirical evaluations of SATM demonstrate its effectiveness across various applications, including in-domain benchmarks and out-of-domain settings. Moreover, its easy-to-implement properties afford flexibility, allowing it to integrate with other advanced sharpness-aware minimizers.
BibTeX
@misc{gao2025enhancing,
  author = {Boyan Gao and Bo Zhao and Shreyank N Gowda and Xingrun Xing and Yibo Yang and Timothy Hospedales and David A. Clifton},
  title = {{Enhancing Generalization via Sharpness-Aware Trajectory Matching for Dataset Condensation}},
  year = {2025},
  eprint = {2502.01865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01865},
}

Loading related content...

Figure