Three-dimensional shape cues affect human and artificial recognition systems differently

Mikayla J. Cutler, Luke Baumel, Joseph Tocco, William Friebel, George K. Thiruvathukal, Nichoals Baker2025 · DOI

TL;DR

This paper investigates how humans and deep neural networks (DNNs) differ in their use of 3D shape cues (shading and shadows) versus texture for object recognition. They found that while both benefit from 3D cues, humans leverage them to infer 3D structure (especially for non-canonical views), whereas DNNs treat them as additional surface cues, with greater gains for canonical views.

What Problem Does This Paper Address?

Understanding how artificial systems like DNNs process visual information is crucial for developing more human-like AI. Previous research has shown that humans are generally biased towards shape for object recognition, while DNNs tend to rely more heavily on texture. However, much of this research has focused on shape defined by an object's external contour. This paper addresses the question of how 3D shape information, specifically shading and attached shadows, influences object recognition in both humans and DNNs, and whether these systems utilize this information in fundamentally different ways. The authors aim to disentangle whether DNNs' texture bias stems from a lack of shape understanding or a different weighting of cues, and how 3D cues might bridge this gap or reveal deeper discrepancies.

Methods

The researchers created a novel dataset of texture-substituted images derived from 3D ShapeNet models. These images were manipulated to include or exclude 3D cues (shading and attached shadows) and were presented from both canonical and noncanonical viewpoints.

They tested three state-of-the-art neural network architectures: ResNet-50, ViT-16, and SWIN. These networks were trained and evaluated on the custom datasets. The study also involved human participants performing similar classification tasks.

Key experimental manipulations included:

  • Recognition Cue: Participants/networks were instructed to identify either the shape or the texture of an object.
  • 3D Cues: Images were either rendered with shading and attached shadows (3D) or without (2D).
  • Viewpoint Canonicality: Images were presented from either a canonical (familiar) or noncanonical (unfamiliar) perspective.

The researchers analyzed classification accuracy and response times. For DNNs, they also examined "texture bias," defined as the proportion of correct texture classifications relative to total correct shape/texture classifications.

Datasets

  • ShapeNet: A dataset of 3D models used to generate the stimuli.
  • Custom Datasets: Approximately 120,000 texture-substituted images were created from ShapeNet models. These datasets were designed to systematically vary the presence of 3D cues (shading, attached shadows) and viewpoint canonicality.
  • 3D Stimuli: Grayscale images of 3D models multiplied with RGB texture images.
  • 2D Stimuli: Binary silhouette images multiplied with texture images, effectively removing 3D cues.

Key Insights and Ideas

  • 3D Cues Reduce DNN Texture Bias: The inclusion of shading and attached shadows significantly reduced the texture bias in DNNs, making them more shape-reliant. This suggests that 3D cues provide valuable shape information that can override texture dominance.
  • Humans Use 3D Cues for Structural Inference (Noncanonical Views): For humans, 3D cues were most beneficial for recognizing objects from noncanonical viewpoints. This aligns with the idea that humans use these cues to infer the object's underlying 3D structure when the external contour is ambiguous.
  • DNNs Treat 3D Cues as Surface Properties (Canonical Views): In contrast, DNNs showed the greatest performance gains from 3D cues when viewing objects from canonical perspectives. This suggests DNNs likely treat shading and shadows as additional surface-level features, similar to texture, rather than as cues for inferring 3D structure.
  • Fundamental Differences in Information Weighting: The study highlights a core difference in how humans and DNNs process visual information. Humans appear to build more robust, structural representations, while DNNs excel at pattern matching based on learned image-level statistics.

Key Results

  • Shape vs. Texture Bias: Without 3D cues, DNNs exhibited an average texture bias of 44%, classifying objects by shape in 35% of trials and texture in 27%. Humans, however, classified objects by shape in 87% of trials and texture in 56%.
  • Impact of 3D Cues on DNNs: With 3D cues, DNNs' shape bias increased, and their shape classification accuracy improved by approximately 11% on average. Texture classification accuracy decreased by about 8%.
  • Impact of 3D Cues on Humans: Human shape recognition performance increased from 87% to 92% with 3D cues. This gain was more pronounced for noncanonical images.
  • Canonical vs. Noncanonical Views:
    • Humans: 3D cues significantly improved performance for noncanonical images (85% to 94% accuracy) but showed little improvement for canonical images (95% vs. 95%).
  • DNNs: All five networks tested showed greater performance gains from 3D information in canonical images compared to noncanonical images.

Limitations and Future Work

  • Task Differences: The authors acknowledge that the neural network and human tests were not perfectly identical (e.g., cue-conflict vs. explicit instruction). However, they argue these differences don't fully explain the observed performance disparities.
  • Generalizability of Finetuning: While finetuning made ViT more human-like in specific tasks, the authors note that this doesn't necessarily generalize to other visual tasks, echoing previous findings that targeted AI training doesn't always translate to broader human-like cognitive abilities.
  • Nature of Shape Representation: The paper discusses two hypotheses: either humans and DNNs form structural representations differently, or neither truly does. The results lean towards the former, suggesting humans might be forming more abstract, configural representations that benefit from 3D cues for structural inference, while DNNs rely more on image-level similarities.
  • Future Directions: The study opens avenues for exploring how DNNs might be trained to develop more human-like structural understanding of shapes, potentially by incorporating curricula that emphasize viewpoint-invariant features or by better simulating human visual diets. Further research could investigate the neural correlates of these different processing strategies.

Summary updated

Abstract

Abstract Humans and neural networks use shape and texture information differently. While shape is the primary cue in human object recognition, neural networks are more biased towards texture cues. Many tests of shape vs. texture bias have focused on shape recognition from an object’s external contour. However, shape information is also conveyed through internal contours, shading, and attached shadows, especially when an object is viewed from noncanonical perspectives. Using models from ShapeNet, we created datasets of 120,000 texture-substituted images of objects from many viewpoints with and without shading and attached shadows. We tested humans’ and several neural networks’ ability to classify these objects by both their shape and their texture. Humans were much better at classifying texture-substituted objects by their shape than any network, although these differences were greater when shape was defined only by the external contour than when 3D cues were included. Our findings suggest that networks’ texture bias is reduced when 3D cues are included in images. We next tested whether the inclusion of 3D cues benefitted humans and neural networks more for images of objects viewed from canonical or noncanonical perspectives. Consistent with earlier research, we found that 3D cues primarily benefitted humans for noncanonical images. For neural networks, the greatest performance gains were for canonical images. These findings suggest fundamental differences in how humans and networks use shading and attached shadows for object recognition. We argue that humans use these cues to infer objects’ 3D structures while neural networks use them as another surface-level cue like texture.
BibTeX
@misc{cutler2025three,
  author = {Mikayla J. Cutler and Luke Baumel and Joseph Tocco and William Friebel and George K. Thiruvathukal and Nichoals Baker},
  title = {{Three-dimensional shape cues affect human and artificial recognition systems differently}},
  year = {2025},
  doi = {10.64898/2025.12.01.691716},
}

Loading related content...

Figure