From Images to 3D Shape Attributes

David F. Fouhey, Abhinav Gupta, Andrew Zisserman

Abstract

This research explores the inference of high-level 3D shape properties from single 2D images. By defining a set of 12 geometric attributes—spanning curvature, contact, and volumetric properties—the authors move beyond traditional metric depth reconstruction. Utilizing a large-scale dataset of over 143,000 sculpture images, they train a Convolutional Neural Network (CNN) to predict these attributes and generate a viewpoint-invariant shape embedding. The results demonstrate that direct attribute inference is more accurate than derivation from depth maps and that these learned geometric features generalize effectively to common object categories.

Introduction: Beyond Metric Depth

Traditional 3D computer vision often prioritizes "reconstruction," aiming to produce per-pixel depth maps or 3D meshes. While useful, this approach faces challenges: depth maps are often noisy, highly dependent on camera viewpoint, and typically category-specific. To address these limitations, Fouhey et al. propose focusing on intrinsic geometric "attributes." These attributes capture fundamental properties of an object's form—such as whether it is hollow, planar, or has thin structures—that remain constant regardless of the camera's position.

Defining the Geometric Vocabulary

The core of this approach is a set of 12 high-level 3D shape attributes. These are categorized into three groups: Curvature Properties (e.g., planarity, cylindricality), Contact Properties (e.g., point vs. multiple contacts), and Volumetric Properties (e.g., presence of holes, thin structures, symmetry).

The 12 defined 3D shape attributes categorized by curvature, contact, and volumetric properties.

By predicting these properties directly, the model avoids the compounding errors inherent in calculating geometric derivatives (like surface normals) from estimated depth maps.

The Sculpture Dataset and Shape Embedding

To train a network capable of understanding pure form, the researchers introduced a massive new dataset of 143,000 images representing 2,197 sculptures. Sculptures provide an ideal training ground because they exhibit immense geometric variety without the functional constraints of everyday objects.

Examples from the sculpture dataset and the viewpoint-invariant matching task using shape embeddings.

The network architecture features two specialized branches. The first predicts the 12 attributes, while the second produces a 1024-dimensional shape embedding. This embedding is trained using a triplet loss to ensure that different views of the same object result in similar vectors, effectively enabling the network to perform a form of "mental rotation" to recognize an object from any angle.

Key Findings and Generalization

The proposed direct inference method achieved a mean AUROC of 72.3% for attribute prediction, significantly outperforming baselines derived from state-of-the-art depth estimation. Furthermore, the shape embedding proved highly robust, achieving 92.3% AUROC in viewpoint-invariant matching tasks.

Crucially, the attributes learned from abstract art generalized to the real world. When tested on the PASCAL VOC dataset, the model correctly characterized common objects: identifying buses as planar and animals as having point-contact with the ground. This suggests that the geometric "vocabulary" learned from sculptures is a fundamental component of general visual perception.

Conclusion

By shifting the focus from metric reconstruction to high-level geometric attributes, this work provides a more robust and viewpoint-invariant way for machines to perceive 3D shape. The success of the sculpture-trained model on everyday objects highlights the power of learning from pure form, opening new avenues for category-agnostic 3D understanding in computer vision.

Summary updated

Abstract

Our goal in this paper is to investigate properties of 3D shape that can be determined from a single image. We define 3D shape attributes-generic properties of the shape that capture curvature, contact and occupied space. Our first objective is to infer these 3D shape attributes from a single image. A second objective is to infer a 3D shape embedding-a low dimensional vector representing the 3D shape. We study how the 3D shape attributes and embedding can be obtained from a single image by training a Convolutional Neural Network (CNN) for this task. We start with synthetic images so that the contribution of various cues and nuisance parameters can be controlled. We then turn to real images and introduce a large scale image dataset of sculptures containing 143K images covering 2197 works from 242 artists. For the CNN trained on the sculpture dataset we show the following: (i) which regions of the imaged sculpture are used by the CNN to infer the 3D shape attributes; (ii) that the shape embedding can be used to match previously unseen sculptures largely independent of viewpoint; and (iii) that the 3D attributes generalize to images of other (non-sculpture) object classes.
BibTeX
@article{fouhey2016from,
  author = {David F. Fouhey and Abhinav Gupta and Andrew Zisserman},
  title = {{From Images to 3D Shape Attributes}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2016},
  url = {https://openreview.net/forum?id=aBfvHEMTp-M},
}
Figure