Approaching human 3D shape perception with neurally mappable models

Thomas P. O'Connell, Tyler Bonnen, Yoni Friedman, Ayush Tewari, Joshua B. Tenenbaum, Vincent Sitzmann, Nancy Kanwisher

Abstract: Standard computer vision models, including state-of-the-art CNNs and Vision Transformers, often fail to perceive 3D shapes with the same robustness as humans, frequently relying on 2D textures instead. This research investigates whether 3D neural fields can bridge this "3D shape processing gap." We show that 3D Light Field Networks (3D-LFNs), when trained with multi-view objectives, can achieve human-level accuracy and error consistency on viewpoint-invariant matching tasks. While these models represent a significant step toward human-like 3D perception, they still face limitations in generalizing to entirely novel object categories.

Introduction: The 3D Shape Processing Gap

Despite the massive success of Deep Neural Networks (DNNs) in object recognition, a persistent gap remains in 3D shape processing. Humans effortlessly recognize an object’s global 3D structure from a single viewpoint and can predict its appearance from other angles. In contrast, standard models trained on ImageNet exhibit a strong "texture bias." When tasked with matching objects across different viewpoints—a task humans find intuitive—standard CNNs and Vision Transformers (ViTs) significantly underperform. This study explores whether the "secret sauce" of 3D perception lies in model architecture, training data, or learning objectives.

Methods and Experimental Design

To quantify the gap between artificial and human vision, the researchers employed a multi-view match-to-sample task. In this paradigm, participants and models are shown a "sample" object and must identify which of two alternatives—a "target" or a "lure"—represents the same object from a different viewpoint.

Figure 1: The experimental paradigm includes a match-to-sample task (a), model evaluation using cosine similarity (b), and the resulting performance gap between humans and standard ImageNet-trained models (c).

The study utilized several key datasets:

  • ShapeNet: Approximately 30,000 3D CAD models across 13 categories (e.g., cars, chairs) for training and testing.
  • ShapeGen: A novel dataset of procedurally generated abstract shapes used to test 3D perception independent of familiar category labels.
  • Adversarial Trials: A "model zoo" of 25 CNNs was used to identify object pairs that are particularly challenging for standard AI, providing a rigorous test for human-model alignment.

The core model under investigation is the 3D Light Field Network (3D-LFN). Unlike standard classifiers, a 3D-LFN learns a continuous function representing an object's light field. The pipeline involves an encoder (ResNet50) extracting shape latents, a hypernetwork mapping these to neural field weights, and a renderer that queries the field to generate views from arbitrary camera positions.

Results: Bridging the Gap with 3D-LFNs

The results indicate that 3D-LFNs significantly outperform standard models, reaching approximately 93% accuracy on within-category matching tasks—surpassing the average human performance of ~89% and falling within the "noise ceiling" of human behavior. This means the model's error patterns are as consistent with humans as humans are with each other.

Figure 2: The 3D-LFN architecture (a) uses an encoder-renderer pipeline to learn 3D representations. The resulting performance (b) shows that 3D-LFN representations (specifically weights) align closely with human behavior, bridging the gap left by standard CNNs and ViTs.

Key insights from the analysis include:

  • Multi-view Training is Crucial: The learning objective proved more important than the specific architecture. Standard CNNs modified with a multi-view objective saw a significant jump in human alignment (~83% accuracy).
  • Analysis-by-Synthesis: The generative nature of 3D-LFNs, which allows them to "visualize" internal 3D representations, appears to mirror human mental imagery more closely than simple discriminative classification.
  • Robustness to Geometry: Both humans and 3D-LFNs remained accurate on "adversarial" trials that fooled ImageNet models, suggesting a shared reliance on underlying 3D structural features.

Discussion and Limitations

While 3D-LFNs represent a major advance, they still face a "generalization barrier." Humans can perceive the 3D form of a totally novel sculpture, but 3D-LFN performance drops significantly when tested on categories not seen during training (e.g., training on cars/chairs and testing on planes).

Furthermore, the current models rely on ground-truth camera matrices during training, which is biologically implausible. Future research should explore how models might learn a "universal grammar of 3D shape" through unsupervised learning from video or ego-motion data, more closely mimicking the developmental process of human infants.

Summary updated

BibTeX
@inproceedings{oconnell2025approaching,
  author = {Thomas P. O'Connell and Tyler Bonnen and Yoni Friedman and Ayush Tewari and Joshua B. Tenenbaum and Vincent Sitzmann and Nancy Kanwisher},
  title = {{Approaching human 3D shape perception with neurally mappable models}},
  booktitle = {Open},
  year = {2025},
  doi = {10.1162/opmi\_a\_00189},
}
Figure