TL;DR
This paper introduces OoD-ViT-NAS, the first systematic benchmark designed to evaluate how Vision Transformer (ViT) architectures perform under Out-of-Distribution (OoD) shifts. By testing 3,000 unique architectures across 8 large-scale OoD datasets, the authors demonstrate that standard In-Distribution (ID) accuracy is a poor predictor of robustness, and that scaling up a model's embedding dimension is the single most effective architectural lever for boosting real-world generalization.
What Problem Does This Paper Address?
While Vision Transformers (ViTs) excel at standard computer vision tasks when the test data looks exactly like the training data (In-Distribution), they often struggle to generalize when faced with real-world variations like blur, weather changes, or stylistic shifts (Out-of-Distribution).
Currently, both manual architecture design and automated Neural Architecture Search (NAS) focus almost entirely on maximizing ID accuracy. We lack a systematic understanding of how structural choices—like network depth, embedding dimensions, or the number of attention heads—impact a model's resilience to distribution shifts. Furthermore, evaluating thousands of architectures is incredibly expensive; training 3,000 ViTs from scratch would require prohibitive amounts of compute and time. This paper introduces a massive, compute-efficient benchmark to bridge this gap and provide actionable insights into robust ViT design.
Methods
To bypass the extreme computational cost of training thousands of models, the authors leverage One-Shot NAS via the AutoFormer search space. Instead of training 3,000 models individually, they train three "supernets" (Tiny, Small, and Base). They then randomly sample 1,000 sub-architectures (subnets) from each supernet. These subnets inherit their weights directly from the pre-trained supernets, allowing for instant evaluation without additional training.
Using this benchmark, the authors conduct three core investigations:
- ID vs. OoD Correlation: Quantifying how well standard validation performance tracks with robustness.
- Training-free NAS Evaluation: Testing 9 popular zero-cost proxies (e.g., SNIP, Grasp, SynFlow, AutoProx) to see if they can predict a model's OoD performance at initialization without any training.
- Structural Attribute Analysis: Isolating individual architectural parameters to determine their specific impact on robustness.
Datasets
The benchmark evaluates performance using ImageNet-1K as the In-Distribution baseline, alongside 8 prominent large-scale OoD datasets representing different types of distribution shifts:
- Algorithmic Corruptions: ImageNet-C (15 corruption types like blur and noise across 5 severities), ImageNet-P (perturbation sequences), and ImageNet-D (diffusion-synthetic objects).
- Natural & Stylistic Shifts: ImageNet-A (natural adversarial examples), ImageNet-O (open-set anomalies), ImageNet-R (artistic renditions), ImageNet-Sketch (black-and-white sketches), and Stylized ImageNet (images with stripped textures and modified styles).
Key Insights and Ideas
- ID Accuracy is a Deceptive Guide: Squeezing out an extra 0.5% accuracy on standard ImageNet validation does not guarantee a more robust model. The correlation between ID and OoD accuracy is surprisingly weak, meaning architectures optimized purely for standard datasets risk being fragile in production.
- Complex Proxies Fail, Simplicity Wins: Zero-cost proxies designed to accelerate NAS completely fall apart when predicting OoD performance. Shockingly, simple baselines like total parameter count (
#Param) or floating-point operations (#Flops) consistently outperform all 9 complex training-free NAS methods. - The Dominance of Embedding Dimensions: Among all structural variables, the embedding dimension has the highest positive correlation with OoD robustness.
- Why Embedding Dimensions Matter: Through frequency analysis, the authors discovered that increasing the embedding dimension specifically helps ViTs preserve and learn High-Frequency Components (HFC). Because many common corruptions (like blur) destroy high-frequency details, models with larger embedding dimensions are inherently better equipped to handle these distortions.
Key Results
- The Robustness Gap: Architectures with nearly identical ID accuracy (within a tight 1.39% range) exhibited massive variance in OoD accuracy—diverging by up to 11.85% on specific shifts like pixelation.
- Severity Amplifies Design Flaws: As the severity of an environmental corruption increases, the performance gap between well-designed and poorly-designed architectures widens significantly.
- Smart Scaling Beats Compound Scaling: Human designers typically scale models up using "compound scaling" (increasing depth, heads, and dimensions simultaneously). However, the authors showed that simply increasing the embedding dimension of a standard ViT-B-32 achieved an IN-R OoD accuracy of 48.28%, outperforming the much larger, compound-scaled ViT-L-32 (44.33%) while using a fraction of the parameters and latency.
Limitations and Future Work
While weight-sharing via One-Shot NAS is highly efficient, subnets can occasionally exhibit minor performance discrepancies compared to models trained entirely from scratch. Additionally, this benchmark is heavily anchored within the ImageNet ecosystem; validating these architectural insights on other diverse domains, such as medical imaging or remote sensing, remains an open avenue for future research. Finally, the total failure of existing zero-cost proxies challenges the community to develop new, training-free NAS metrics explicitly designed to predict out-of-distribution generalization.
Summary updated