Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights

Sy-Tuyen Ho, Tuan Van Vo, Somayeh Ebrahimkhani, Ngai-Man Cheung

TL;DR

This paper introduces OoD-ViT-NAS, the first systematic benchmark designed to evaluate how Vision Transformer (ViT) architectures perform under Out-of-Distribution (OoD) shifts. By testing 3,000 unique architectures across 8 large-scale OoD datasets, the authors demonstrate that standard In-Distribution (ID) accuracy is a poor predictor of robustness, and that scaling up a model's embedding dimension is the single most effective architectural lever for boosting real-world generalization.

What Problem Does This Paper Address?

While Vision Transformers (ViTs) excel at standard computer vision tasks when the test data looks exactly like the training data (In-Distribution), they often struggle to generalize when faced with real-world variations like blur, weather changes, or stylistic shifts (Out-of-Distribution).

Currently, both manual architecture design and automated Neural Architecture Search (NAS) focus almost entirely on maximizing ID accuracy. We lack a systematic understanding of how structural choices—like network depth, embedding dimensions, or the number of attention heads—impact a model's resilience to distribution shifts. Furthermore, evaluating thousands of architectures is incredibly expensive; training 3,000 ViTs from scratch would require prohibitive amounts of compute and time. This paper introduces a massive, compute-efficient benchmark to bridge this gap and provide actionable insights into robust ViT design.

Methods

To bypass the extreme computational cost of training thousands of models, the authors leverage One-Shot NAS via the AutoFormer search space. Instead of training 3,000 models individually, they train three "supernets" (Tiny, Small, and Base). They then randomly sample 1,000 sub-architectures (subnets) from each supernet. These subnets inherit their weights directly from the pre-trained supernets, allowing for instant evaluation without additional training.

Using this benchmark, the authors conduct three core investigations:

  1. ID vs. OoD Correlation: Quantifying how well standard validation performance tracks with robustness.
  2. Training-free NAS Evaluation: Testing 9 popular zero-cost proxies (e.g., SNIP, Grasp, SynFlow, AutoProx) to see if they can predict a model's OoD performance at initialization without any training.
  3. Structural Attribute Analysis: Isolating individual architectural parameters to determine their specific impact on robustness.

Datasets

The benchmark evaluates performance using ImageNet-1K as the In-Distribution baseline, alongside 8 prominent large-scale OoD datasets representing different types of distribution shifts:

  • Algorithmic Corruptions: ImageNet-C (15 corruption types like blur and noise across 5 severities), ImageNet-P (perturbation sequences), and ImageNet-D (diffusion-synthetic objects).
  • Natural & Stylistic Shifts: ImageNet-A (natural adversarial examples), ImageNet-O (open-set anomalies), ImageNet-R (artistic renditions), ImageNet-Sketch (black-and-white sketches), and Stylized ImageNet (images with stripped textures and modified styles).

Key Insights and Ideas

  • ID Accuracy is a Deceptive Guide: Squeezing out an extra 0.5% accuracy on standard ImageNet validation does not guarantee a more robust model. The correlation between ID and OoD accuracy is surprisingly weak, meaning architectures optimized purely for standard datasets risk being fragile in production.
  • Complex Proxies Fail, Simplicity Wins: Zero-cost proxies designed to accelerate NAS completely fall apart when predicting OoD performance. Shockingly, simple baselines like total parameter count (#Param) or floating-point operations (#Flops) consistently outperform all 9 complex training-free NAS methods.
  • The Dominance of Embedding Dimensions: Among all structural variables, the embedding dimension has the highest positive correlation with OoD robustness.
Figure 1: We propose, OoD-ViT-NAS, the first comprehensive benchmark for NAS on OoD generalization of ViT architectures. Then, we comprehensively investigate OoD generalization for ViT. The detailed of 8 OoD datasets in our investigation can be found in Tab. 1. In this figure, we show the Kendall τ ranking correlation between OoD accuracy of different datasets on the left and different quantities at the bottom. Our analysis uncovers several key insights. (a) ID as an indicator for ViT OoD Generalization (Sec. 4.2) We show that the correlation between ID accuracy and OoD accuracy is not very high. This suggests that current architectural insights based on ID accuracy might not translate well to OoD generalization. (b) Training-free NAS for ViT OoD Generalization. (Sec. 4.3) We conduct the first study of NAS for ViT’s OoD generalization, showing that their effectiveness significantly weakens in predicting OoD accuracy. (c) OoD Generalization ViT Architectural Attributes. (Sec. 4.4) Our first study on the impact of ViT architectural attributes on OoD generalization shows that the embedding dimension generally has the highest correlation with OoD accuracy among ViT architectural attributes. Additional results can be found in the Appx.
  • Why Embedding Dimensions Matter: Through frequency analysis, the authors discovered that increasing the embedding dimension specifically helps ViTs preserve and learn High-Frequency Components (HFC). Because many common corruptions (like blur) destroy high-frequency details, models with larger embedding dimensions are inherently better equipped to handle these distortions.

Key Results

  • The Robustness Gap: Architectures with nearly identical ID accuracy (within a tight 1.39% range) exhibited massive variance in OoD accuracy—diverging by up to 11.85% on specific shifts like pixelation.
  • Severity Amplifies Design Flaws: As the severity of an environmental corruption increases, the performance gap between well-designed and poorly-designed architectures widens significantly.
  • Smart Scaling Beats Compound Scaling: Human designers typically scale models up using "compound scaling" (increasing depth, heads, and dimensions simultaneously). However, the authors showed that simply increasing the embedding dimension of a standard ViT-B-32 achieved an IN-R OoD accuracy of 48.28%, outperforming the much larger, compound-scaled ViT-L-32 (44.33%) while using a fraction of the parameters and latency.

Limitations and Future Work

While weight-sharing via One-Shot NAS is highly efficient, subnets can occasionally exhibit minor performance discrepancies compared to models trained entirely from scratch. Additionally, this benchmark is heavily anchored within the ImageNet ecosystem; validating these architectural insights on other diverse domains, such as medical imaging or remote sensing, remains an open avenue for future research. Finally, the total failure of existing zero-cost proxies challenges the community to develop new, training-free NAS metrics explicitly designed to predict out-of-distribution generalization.

Summary updated

Abstract

While ViTs have achieved across machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under OoD shifts. A crucial research gap exists in understanding how to design ViT architectures, both manually and automatically, for better OoD generalization. To this end, we introduce OoD-ViT-NAS, the first systematic benchmark for ViTs NAS focused on OoD generalization. This benchmark includes 3000 ViT architectures of varying computational budgets evaluated on 8 common OoD datasets. Using this benchmark, we analyze factors contributing to OoD generalization. Our findings reveal key insights. First, ViT architecture designs significantly affect OoD generalization. Second, ID accuracy is often a poor indicator of OoD accuracy, highlighting the risk of optimizing ViT architectures solely for ID performance. Third, we perform the first study of NAS for ViTs OoD robustness, analyzing 9 Training-free NAS methods. We find that existing Training-free NAS methods are largely ineffective in predicting OoD accuracy despite excelling at ID accuracy. Simple proxies like Param or Flop surprisingly outperform complex Training-free NAS methods in predicting OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization and discover that increasing embedding dimensions generally enhances performance. Our benchmark shows that ViT architectures exhibit a wide range of OoD accuracy, with up to 11.85% improvement for some OoD shifts. This underscores the importance of studying ViT architecture design for OoD. We believe OoD-ViT-NAS can catalyze further research into how ViT designs influence OoD generalization.
BibTeX
@inproceedings{ho2024vision,
  author = {Sy-Tuyen Ho and Tuan Van Vo and Somayeh Ebrahimkhani and Ngai-Man Cheung},
  title = {{Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=2AIwiIkE0s},
}
Figure