Progress and limitations of deep networks to recognize objects in unusual poses

Amro Abbas, Stéphane Deny

Abstract

Deep networks should be robust to rare events if they are to be successfully deployed in high-stakes real-world applications (e.g., self-driving cars). This study investigates the capability of 38 recent and competitive deep networks to recognize objects in unusual poses using a new synthetic dataset, ObjectPose. While classifying these images remains a challenge—with an average accuracy drop of 29.5% compared to upright objects—scaling training data significantly improves robustness. The best-performing model, Noisy Student EfficientNet-L2, showed a 14.5% drop. However, visual inspection reveals that even the most robust models still fail in ways that differ fundamentally from the human visual system.

Introduction: The Challenge of Geometric Robustness

While deep neural networks (DNNs) have famously surpassed human performance on standard image benchmarks like ImageNet, their reliability in real-world scenarios remains a concern. One major hurdle is "unusual poses"—how does a vision system react when it sees a bus flipped on its side or a forklift viewed from directly underneath? Humans are remarkably good at "mental rotation," recognizing objects regardless of their orientation. In contrast, DNNs, which are largely feed-forward and trained on "upright" bias, often fail when the geometry shifts.

Methodology and the ObjectPose Dataset

To quantify this "brittleness," researchers developed ObjectPose, a custom synthetic dataset of 27,540 images featuring 17 object categories (e.g., 'airliner', 'hammerhead shark', 'tank') rendered in every possible orientation against various backgrounds. By using 3D models, they precisely controlled rotations along three axes (Yaw, Roll, Pitch).

Examples of objects in unusual poses from the ObjectPose dataset, showcasing the variety of orientations and backgrounds used to test model robustness.

The study evaluated 38 competitive models across various architectures, including classic CNNs (ResNet), Vision Transformers (ViT), MLP-Mixers, and "foundation models" like CLIP and Noisy Student EfficientNet. The researchers also compared "in-plane" rotation (2D) vs. "out-of-plane" rotation (3D) and tested combinatorial stress by combining rotation with scaling.

Key Results: The Scaling Law of Robustness

The results reveal a significant "accuracy cliff." On average, the models tested saw a 29.5% drop in accuracy when moving from upright to unusual poses.

Summary of results: (A) Accuracy of 38 models on ObjectPose compared to ImageNet (IN) accuracy. (B) ObjectPose accuracy vs. ObjectPose +-10 (usual pose) accuracy, where bubble size indicates training dataset size.

The single biggest predictor of pose robustness was not architecture (CNN vs. Transformer), but the size of the training set. Models trained on massive datasets like JFT-300M or 3.6 billion Instagram images were significantly more resilient. The Noisy Student EfficientNet-L2 (trained on JFT-300M) was the top performer, with only a 14.5% drop. This suggests that while architectural choices like Transformers vs. CNNs have minimal impact on this specific robustness, massive data scaling is a viable, if expensive, path forward.

Insights into Model Strategies and Failures

  • Background Reliance: Weak models often rely on an upright background to recognize objects. Stronger models, however, are actually distracted by an upright background when the object is tilted, performing better when the background is removed entirely.
  • Multiplicative Errors: Combining transformations (e.g., rotation plus scaling) further degrades performance, following a combinatorial failure model where each transformation adds an independent chance for the model to fail.
  • The "Silly" Failure Gap: Even the best-performing models make errors a human never would—for example, confusing a 90°-rotated cannon with a "guillotine" or a "harp." This suggests that current models lack a true 3D understanding of object geometry.

Conclusion and Future Directions

While modern deep networks are closing the gap through massive scaling, unusual object orientations still trigger significant performance hits. The findings held up on real-world data (CO3D), though the drop was less severe, likely due to photographer bias toward "usual" poses.

The most provocative takeaway is the potential need for Recurrent Mechanisms. Humans take longer to recognize objects the more they are rotated, suggesting a mental rotation process. DNNs, being feed-forward, take the same amount of time regardless of pose. Adding recurrent "loops" to AI architectures might be the necessary step to achieve human-level geometric common sense and true robustness in the real world.

Summary updated

Abstract

Deep networks should be robust to rare events if they are to be successfully deployed in high-stakes real-world applications (e.g., self-driving cars). Here we study the capability of deep networks to recognize objects in unusual poses. We create a synthetic dataset of images of objects in unusual orientations, and evaluate the robustness of a collection of 38 recent and competitive deep networks for image classification. We show that classifying these images is still a challenge for all networks tested, with an average accuracy drop of 29.5% compared to when the objects are presented upright. This brittleness is largely unaffected by various network design choices, such as training losses (e.g., supervised vs. self-supervised), architectures (e.g., convolutional networks vs. transformers), dataset modalities (e.g., images vs. image-text pairs), and data-augmentation schemes. However, networks trained on very large datasets substantially outperform others, with the best network tested$\unicode{x2014}$Noisy Student EfficentNet-L2 trained on JFT-300M$\unicode{x2014}$showing a relatively small accuracy drop of only 14.5% on unusual poses. Nevertheless, a visual inspection of the failures of Noisy Student reveals a remaining gap in robustness with the human visual system. Furthermore, combining multiple object transformations$\unicode{x2014}$3D-rotations and scaling$\unicode{x2014}$further degrades the performance of all networks. Altogether, our results provide another measurement of the robustness of deep networks that is important to consider when using them in the real world. Code and datasets are available at https://github.com/amro-kamal/ObjectPose.
BibTeX
@inproceedings{abbas2023progress,
  author = {Amro Abbas and Stéphane Deny},
  title = {{Progress and limitations of deep networks to recognize objects in unusual poses}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=UUy3OSSdAC},
}

Loading related content...

Figure