Why Distilled Datasets Overfit to Specific Neural Architectures

290 concepts184 papers

Dataset distillation aims to compress massive datasets into a tiny handful of highly informative synthetic samples. Imagine shrinking the entire ImageNet dataset down to just ten images per class, yet still being able to train a neural network on it to high accuracy. While this promise holds immense potential for accelerating network training and architecture search, it faces a critical roadblock: architecture overfitting.

When a synthetic dataset is distilled using a specific neural network architecture (the surrogate or teacher model), it becomes highly specialized. If you attempt to train a different network architecture (the student or target model) on these same synthetic images, performance frequently plummets. This cross-architecture generalization gap severely limits the portability of distilled data.

The Root Cause: Adversarial Shortcuts and Inductive Biases

Why does this generalization gap exist? To understand it, we must look at how dataset distillation algorithms operate. Most traditional methods optimize the pixels of synthetic images by forcing a surrogate model's training trajectory, gradients, or features to match those of the real dataset.

During this optimization loop, the synthetic data is free to change in any way that satisfies the objective function. Because neural networks are notorious shortcut learners, the optimization process quickly discovers high-frequency patterns that perfectly trigger the surrogate model's specific architectural components. As Antonio Torralba observes in Generalizing Dataset Distillation via Deep Generative Prior, without a strong prior, distillation algorithms capture high-frequency noise that serves as a model-specific shortcut rather than generalizable semantic features.

Consequently, these synthetic images end up looking less like real-world objects and more like adversarial noise. As George Cazenavette et al. point out in Dataset Distillation for Pre-Trained Self-Supervised Vision Models, this noise overfits to the specific quirks of the training architecture, making the distilled datasets functionally adversarial to other models. The synthetic data becomes deeply coupled to the inductive biases—such as the translation equivariance of a convolutional neural network (CNN)—of the source model, a phenomenon documented by Wei Jin et al. in Graph Condensation for Graph Neural Networks and Noel Loo et al. in Large Scale Dataset Distillation with Domain Shift.

The Architectural Divide

This coupling creates severe friction when transferring data across different model families. The most prominent manifestation occurs when moving between convolutional networks and attention-based models. As highlighted by Samir Khaki et al. in ATOM: Attention Mixer for Efficient Dataset Distillation and Ahmad Sajedi et al. in Data-to-Model Distillation: Data-Efficient Learning Framework, datasets distilled using CNNs exhibit a persistent performance drop when used to train Vision Transformers (ViTs). A ViT lacks the hardcoded spatial inductive biases of a CNN, meaning it cannot exploit the same structural shortcuts embedded in the synthetic pixels.

This phenomenon is not restricted to computer vision:

  • Graphs: In graph representation learning, Qingyun Sun et al. demonstrated in GC-Bench: An Open and Unified Benchmark for Graph Condensation that synthetic graphs generated using simple message-passing backbones fail to perform well when transferred to complex Graph Transformers. The reduction process often overlooks the long-range global dependencies that advanced models rely on.
  • Text: In natural language processing, Yefan Tao et al. note in Textual Dataset Distillation via Language Model Embedding that textual distillation often produces abstract mathematical vectors that are rigidly locked to a specific model architecture or tokenizer.

Mitigating the Gap: Designing for Portability

To break this architecture lock-in, researchers are shifting away from direct pixel-and-trajectory matching toward methods that enforce architectural agnosticism.

Distribution Matching

Instead of forcing a synthetic dataset to mimic the exact, chaotic training trajectory of a specific network, methods like those proposed by Bo Zhao et al. in Dataset Condensation with Distribution Matching match the feature distributions of real and synthetic data across randomly sampled embedding spaces. This abstracts the optimization away from a single model's step-by-step learning path.

Generative Latent Priors

Rather than optimizing raw pixels directly, frameworks like Generative Latent Distillation (GLaD), developed by Antonio Torralba, optimize the data within the latent space of a pre-trained generative model. By restricting the synthetic data to a natural image manifold, it prevents the formation of high-frequency adversarial noise, forcing the distilled data to express robust, transferable semantic concepts. Yuqi Jia et al. similarly leverage deep generative latents to ensure architecture agnosticism in federated environments in Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents.

Complexity and Smoothing Regularization

Other approaches introduce explicit regularizers to prevent overfitting during distillation. For instance, Chen Liu et al. utilize architectural smoothing techniques like DropPath and knowledge distillation from small teachers in Towards Mitigating Architecture Overfitting in Dataset Distillation to prevent the synthetic data from adapting to a single network depth or topology.

From an information-theoretic perspective, Xinhao Zhong introduced Class-aware Conditional Mutual Information (CMI) in Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information as a plug-and-play penalty to constrain the class-aware complexity of synthetic data, preventing it from encoding model-specific shortcuts.

By shifting the distillation objective from mimicking a specific model's behavior to capturing the underlying data distribution, the field is moving closer to creating truly universal synthetic proxies.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 191 papers
  2. Version 1 · 2026-10-04 · 264 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_cross_architecture_generalization_gap_e920e673,
  title = {cross-architecture generalization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-architecture-generalization-gap/rdr_e920e67334eb4bb788ac8e71968ca653},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_cross_architecture_generalization_gap_e920e673_v2,
  title = {cross-architecture generalization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-architecture-generalization-gap/rdr_e920e67334eb4bb788ac8e71968ca653/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake