Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations

Avi Cooper, Xavier Boix, Daniel Harari, Spandan Madan, Hanspeter Pfister, Tomotake Sasaki, Pawan Sinha

Abstract

The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the distribution of the training data is not well understood. We present evidence that DNNs are capable of generalizing to objects in novel orientations by disseminating orientation-invariance obtained from familiar objects seen from many viewpoints. This capability strengthens when training the DNN with an increasing number of familiar objects, but only in orientations that involve 2D rotations of familiar orientations. We show that this dissemination is achieved via neurons tuned to common features between familiar and unfamiliar objects. These results implicate brain-like neural mechanisms for generalization.

Introduction

While Deep Neural Networks (DNNs) have achieved human-level performance in many computer vision tasks, they remain notoriously brittle when encountering "out-of-distribution" (OoD) data. A primary failure mode is orientation: a model trained to recognize an object from a specific set of angles often fails when that same object is presented from a novel viewpoint. In contrast, biological systems exhibit robust viewpoint invariance. If a human sees a novel airplane from the front, they can easily recognize it from the side by leveraging prior experience with other airplane models. This study investigates whether DNNs possess a similar "brain-like" mechanism for transferring viewpoint knowledge across different instances within an object category.

Experimental Methodology

To investigate this, the researchers employed a "Fully-seen vs. Partially-seen" training paradigm. In this setup, certain object instances (e.g., specific airplane models) are designated as "fully-seen" and presented to the network from all possible 3D orientations during training. Other instances are "partially-seen," shown only from a limited "seed" range of orientations. The core test is whether the network can recognize these partially-seen instances in orientations it has never encountered for those specific objects.

The experimental paradigm: Dissemination of orientation invariance from fully-seen to partially-seen object instances.

The study utilized three primary datasets totaling 200,000 images: airplanes and cars curated from ShapeNet, and Shepard & Metzler (SM) objects—highly asymmetric 3D shapes used in mental rotation psychology tasks. Standard architectures, including ResNet18, DenseNet, and the brain-inspired CORnet, were evaluated. To quantify generalization, the researchers developed per-orientation accuracy heatmaps, discretizing the 3D rotation space (Euler angles) into "cubelets."

Key Findings

The Dissemination of Orientation Invariance

The results demonstrate that generalization is "disseminated": invariance learned from fully-seen objects is transferred to partially-seen objects. This capability is highly dependent on data diversity. As the number of fully-seen instances in the training set increased from 10 to 40, OoD accuracy for partially-seen instances rose significantly. This suggests that increasing the variety of objects seen from multiple angles is more effective for generalization than simply increasing the total number of images for a few objects.

The In-Plane Rotation Bias

Generalization is not uniform across all 3D rotations. DNNs are remarkably proficient at generalizing to in-plane (2D) rotations—similar to a clock hand spinning—where the object's silhouette remains relatively consistent. However, they struggle with complex 3D rotations that involve significant self-occlusion (where parts of the object disappear behind others). While accuracy in generalizable orientations was well above chance, performance in non-generalizable 3D orientations often remained near chance levels (2-10%).

Classification accuracy heatmaps showing the effect of training diversity and orientation on generalization.

Neuronal Mechanisms

Neuronal analysis of the penultimate layer revealed that individual neurons act as shared feature detectors. These neurons become tuned to specific parts (e.g., a wing or a wheel) shared across different instances. Because these neurons develop orientation invariance through exposure to fully-seen examples, they provide "free" invariance when those same features appear in novel objects. The researchers found a tight correlation between a neuron's "invariance score" and the network's overall classification accuracy.

Discussion and Future Work

The study identifies a significant "3D bottleneck" in current DNN architectures. While they can disseminate 2D rotation logic, they lack the robust 3D mental rotation capabilities of humans. A predictive model of these generalization rules achieved a Pearson correlation coefficient > 0.8 with actual DNN behavior, confirming that these networks rely heavily on shared feature similarity and in-plane rotation logic.

Future research may explore how biological agents overcome these limits, perhaps by leveraging the temporal continuity of video to learn 3D invariance. Additionally, exploring 3D-aware architectures or alternative sensing approaches may help bridge the gap between artificial and biological vision systems.

Summary updated

Abstract

The capability of Deep Neural Networks (DNNs) to recognize objects in orientations outside the distribution of the training data is not well understood. We present evidence that DNNs are capable of generalizing to objects in novel orientations by disseminating orientation-invariance obtained from familiar objects seen from many viewpoints. This capability strengthens when training the DNN with an increasing number of familiar objects, but only in orientations that involve 2D rotations of familiar orientations. We show that this dissemination is achieved via neurons tuned to common features between familiar and unfamiliar objects. These results implicate brain-like neural mechanisms for generalization.
BibTeX
@misc{cooper2021emergent,
  author = {Avi Cooper and Xavier Boix and Daniel Harari and Spandan Madan and Hanspeter Pfister and Tomotake Sasaki and Pawan Sinha},
  title = {{Emergent Neural Network Mechanisms for Generalization to Objects in Novel Orientations}},
  year = {2021},
  eprint = {2109.13445},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13445},
}
Figure