Abstract
Recent state-of-the-art vision models introduced new architectures, learning paradigms, and larger pretraining data, leading to impressive performance on tasks such as classification. While previous generations of vision models were shown to lack robustness to factors such as pose, it's unclear the extent to which this next generation of models are more robust. To study this question, we develop a dataset of more than 7 million images with controlled changes in pose, position, background, lighting, and size. We study not only how robust recent state-of-the-art models are, but also the extent to which models can generalize variation in factors when they're present during training. We consider a catalog of recent vision models, including vision transformers (ViT), self-supervised models such as masked autoencoders (MAE), and models trained on larger datasets such as CLIP. We find out-of-the-box, even today's best models are not robust to common changes in pose, size, and background. When some samples varied during training, we found models required a significant portion of diversity to generalize -- though eventually robustness did improve. When diversity is only seen for some classes however, we found models did not generalize to other classes, unless the classes were very similar to those seen varying during training. We hope our work will shed further light on the blind spots of SoTA models and spur the development of more robust vision models.
Introduction
In recent years, computer vision has seen rapid progress, with models frequently achieving state-of-the-art (SoTA) results on standard benchmarks like ImageNet. However, high leaderboard accuracy does not necessarily equate to robust understanding. Researchers have long suspected that these models may rely on specific textures or contexts—acting as "clever Hans" learners—rather than a deep geometric understanding of objects. While it is known that models struggle with out-of-distribution data, a systematic quantification of why they fail has been missing. This paper addresses this gap by mapping the robustness limits of modern architectures, including Vision Transformers (ViTs) and self-supervised learners, against a comprehensive catalog of natural variations.
Methodology
The authors evaluate a broad catalog of SoTA models, including CNNs (ResNet), ViTs, and MLP-Mixers, across various learning paradigms such as supervised learning, self-supervised learning (SSL) (e.g., MAE, SimCLR, iBot), and multimodal pretraining (CLIP). Evaluation is performed using both linear probing (frozen backbone) and full finetuning.
To facilitate this study, a massive controlled dataset of ~7 million images was created. Using 3D models of 54 object categories from the 3D Warehouse (Trimble Inc), the authors rendered images with precise control over:
- Scalar factors: 101 values for pose, position, size, and lighting color.
- Backgrounds: 25 high-quality natural backgrounds (e.g., sky, water, city).
The dataset's structure—comprising single-factor, paired-factor, and all-factor variations—allows for isolating the exact causes of model failure, a task nearly impossible with "in-the-wild" datasets.
Key Results and Insights
The study reveals that even the most advanced models possess significant "blind spots."
The "Big Three" Weaknesses
Models are most sensitive to changes in pose, background, and size. Conversely, they show relative robustness to position (translation) and lighting color. This is likely because position and lighting are well-approximated by standard data augmentations like cropping and color jitter, whereas pose and 3D size require a deeper geometric understanding.
As illustrated above, changing an object's pose can cause a staggering ~50% drop in accuracy on average. Background changes also result in a significant ~22% drop.
Learning Paradigms and Generalization Gaps
The research highlights a complex relationship between learning paradigms and robustness. While finetuning improves "canonical" accuracy, it often hurts robustness compared to linear evaluation, as models may over-specialize on the training distribution.
Interestingly, supervised models generally benefit more from finetuning for robustness, whereas SSL models (like MAE) often see their inherent robustness advantages eroded by standard finetuning.
The Category Barrier
A striking finding is that robustness does not easily transfer across classes. If a model learns that "cars" can appear in different poses, it does not automatically generalize this concept to "dogs." Generalization only occurs between semantically similar classes, indicating that models struggle to learn the abstract concept of variation.
Conclusion and Future Work
Current SoTA vision models remain surprisingly brittle to natural variations in pose, size, and background. While larger pretraining datasets (e.g., ImageNet-21k) improve resilience, sheer data volume is an inefficient solution. Future work should focus on architectural or algorithmic innovations that help models generalize the concept of variation across the entire label space, moving beyond the need for exhaustive data coverage for every object category.
Summary updated