Dataset distillation aims to compress a massive dataset into a tiny, synthetic proxy. A naive expectation might be that a synthetic dataset optimized to train a specific neural network would overfit to that model's unique inductive biases. If you distill a dataset using a simple convolutional network, why should it successfully train a Vision Transformer or a deep ResNet?
Yet, a recurring discovery across dataset distillation research is that these synthetic summaries possess remarkable cross-architecture transferability. When distillation algorithms target fundamental properties of the data distribution rather than specific model weights, the resulting synthetic samples capture universal structural features. This article explores the core mechanisms that explain why distilled data generalizes across radically different architectures.
Matching Intrinsic Data Distributions
The earliest intuition for cross-architecture transfer comes from the concept of distribution matching. Instead of forcing a synthetic image to maximize the performance of a specific model instance, distillation can force the synthetic data to mimic the statistical footprint of the real dataset.
In their foundational work on gradient matching, Bo Zhao et al. in Dataset Condensation with Gradient Matching demonstrated that optimizing synthetic pixels to produce the same network gradients as real data yields "super-informative" images. Because gradients reflect the underlying loss landscape of the data, this matching process captures structural properties inherent to the data itself. To ensure these transformations remain robust, Bo Zhao et al. introduced Dataset Condensation with Differentiable Siamese Augmentation, showing that applying coordinated, differentiable transformations prevents the synthetic data from exploiting model-specific shortcuts.
To completely decouple distillation from specific model architectures, researchers have turned to explicit statistical alignment. For instance, Hansong Zhang et al. proposed M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy, which aligns all statistical moments of the real and synthetic data distributions within a Reproducing Kernel Hilbert Space. Similarly, Shaobo Wang et al. developed Dataset Distillation with Neural Characteristic Function: A Minmax Perspective, matching the Fourier transform of data distributions. By optimizing the data's global characteristic function, the synthetic data captures universal semantic features that remain highly effective when transitioning from simple convolutional networks to complex, unseen architectures.
Aligning Optimization Trajectories and Attention Maps
Another reason distilled datasets transfer well is that they can encode the temporal dynamics of learning. Rather than matching a model's state at a single snapshot, distillation can match how models evolve over time.
In Dataset Distillation by Matching Training Trajectories, George Cazenavette et al. forced models trained on synthetic data to track the exact parameter trajectory of models trained on the full real dataset. Because diverse neural networks often learn fundamental visual features in a similar order (such as edges and textures before complex semantic parts), matching these trajectories embeds generalizable learning pathways into the synthetic data. This trajectory-matching paradigm generalizes remarkably well; for example, Yuchen Zhang demonstrated in Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching that an adaptive expanding window for trajectory matching allows synthetic graphs to generalize across unseen model structures.
Alternatively, instead of tracking parameters, distillation can target spatial attention. Ahmad Sajedi et al. introduced DataDAM: Efficient Dataset Distillation with Attention Matching, which aligns the spatial attention maps of real and synthetic images across randomly initialized networks. Because spatial attention highlights where informative features reside in an image, focusing on attention maps ensures that the synthetic data captures architecture-agnostic visual salience rather than model-specific weight configurations.
Grounding in Generative Priors and Foundation Models
When synthetic datasets are optimized pixel-by-pixel, they often degenerate into high-frequency, uninterpretable noise. While this noise may satisfy the surrogate model, it fails to transfer to models with different inductive biases. To solve this, recent approaches constrain synthetic data to a realistic generative manifold.
George Cazenavette et al. addressed this in Generalizing Dataset Distillation via Deep Generative Prior by distilling data directly into the latent space of a pre-trained generative model. By ensuring the synthetic data remains visually coherent and structurally sound, the distilled features are grounded in real-world geometry, allowing them to train radically different models like Vision Transformers. Similarly, Jianyang Gu et al. leveraged latent diffusion models in Efficient Dataset Distillation via Minimax Diffusion to ensure that surrogate datasets capture fundamental data characteristics rather than architecture-specific artifacts.
Leveraging foundation models also mitigates architectural bias in the label space. In Heavy Labels Out! Dataset Distillation with Label Space Lightening, Ruonan Yu et al. replaced static soft labels with a lightweight, CLIP-based image-to-label projector. The authors found that labels generated through foundation model priors exhibit significantly higher cross-architecture transferability, particularly when moving between convolutional and transformer-based models, because they leverage universal visual-textual representations.
Universality Across Modalities and Domains
The fact that cross-architecture transferability works is not a phenomenon unique to standard computer vision. It reflects a deeper truth about data synthesis: well-distilled data captures the intrinsic manifold of the task domain.
- Graphs: In Does Graph Distillation See Like Vision Dataset Counterpart?, Beining Yang et al. showed that aligning the spectral properties of synthetic graphs with their original counterparts reduces the performance gap in cross-architecture testing, proving that spectral fidelity is essential for model-agnostic graph summaries.
- 3D Point Clouds: Jae-Young Yim et al. extended this to spatial data in Dataset Distillation of 3D Point Clouds via Distribution Matching, demonstrating that features distilled using a simple backbone remain highly effective for training complex, diverse 3D architectures.
- Multimodal & Text: For vision-language tasks, Yongmin Lee et al. developed CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder, showing that aligning cross-covariance captures fundamental semantic relationships that serve as a general-purpose proxy for diverse multimodal models. Even in pure text settings, Erkun Yang et al. demonstrated in In-context Learning Demonstration Generation with Text Distillation that distilled task demonstrations generalize seamlessly across entirely different Large Language Models.
Summary
Ultimately, cross-architecture transferability works because dataset distillation acts as an advanced form of lossy data compression. Whether through curriculum selection (Gongwei Chen et al.), representative matching (Yanqing Liu), or non-critical region refinement (as explored in Enhancing Dataset Distillation via Non-Critical Region Refinement), the goal is to strip away redundant information and model-specific noise. What remains are the core geometric boundaries, statistical moments, and semantic relationships of the data manifold—features that any capable neural architecture can exploit to learn the underlying task.