Understanding Higher-Order Shape via 3D Shape Attributes

David F. Fouhey, Abhinav Gupta, Andrew Zisserman

Abstract

Our goal in this paper is to investigate properties of 3D shape that can be determined from a single image. We define 3D shape attributes -- generic properties of the shape that capture curvature, contact and occupied space. Our first objective is to infer these 3D shape attributes from a single image. A second objective is to infer a 3D shape embedding -- a low dimensional vector representing the 3D shape. We study how the 3D shape attributes and embedding can be obtained from a single image by training a Convolutional Neural Network (CNN) for this task. We start with synthetic images so that the contribution of various cues and nuisance parameters can be controlled. We then turn to real images and introduce a large scale image dataset of sculptures containing 143K images covering 2197 works from 242 artists. For the CNN trained on the sculpture dataset we show the following: (i) which regions of the imaged sculpture are used by the CNN to infer the 3D shape attributes; (ii) that the shape embedding can be used to match previously unseen sculptures largely independent of viewpoint; and (iii) that the 3D attributes generalize to images of other (non-sculpture) object classes.

Introduction

Traditional 3D computer vision often focuses on "0th-order" properties, such as per-pixel depth maps. While depth maps provide a literal representation of surface distance, they are view-dependent and often lack the high-level semantic "top-down" understanding humans possess—for instance, recognizing that a chair is composed of thin tubes or that a surface is planar. Furthermore, deriving higher-order properties like curvature from noisy, low-resolution depth maps is notoriously error-prone. This research proposes a paradigm shift: predicting 3D shape attributes directly from 2D images to capture the fundamental "character" of a shape.

Methodology: Defining 3D Shape Attributes

The authors define a set of 12 binary 3D shape attributes that capture the geometric essence of an object. These are categorized into three primary groups:

  1. Curvature Properties: Planarity, No Planarity, Cylindrical, and Roughness.
  2. Contact Properties: Point/Line contact with the ground and Multiple contacts.
  3. Volumetric/Topological Properties: Mainly Empty, Multiple Pieces, Has Hole, Thin Structures, Mirror Symmetry, and Cubic Aspect Ratio.
The 12 3D shape attributes categorized by curvature, contact, and volumetric properties.

To infer these, a multi-task Convolutional Neural Network (CNN) based on the VGG-M architecture was employed. The network features two specialized branches: an Attribute Branch that predicts the 12 attributes using independent sigmoids, and an Embedding Branch that projects images into a 1024D vector space using a triplet loss. This ensures that different viewpoints of the same shape are clustered together while distinct shapes are separated.

The Sculpture Dataset

A significant contribution of this work is the introduction of a large-scale image dataset of modern sculptures. Containing 143,000 images of 2,197 works, this dataset serves as a "geometric teacher." Unlike standard object datasets where categories like "bus" have highly predictable shapes, sculptures represent "pure shape." An artist might render a human figure as a series of flat planes or a single smooth curve, forcing the model to learn geometry rather than category-specific textures or silhouettes.

The Sculpture Dataset provides a diverse range of geometric forms and multiple viewpoints for each work, such as Henry Moore's 'Large Two Forms'.

Key Results and Generalization

The direct inference approach significantly outperformed depth-based baselines, achieving a mean AUROC of 72.3% for attribute prediction. The embedding branch demonstrated remarkable robustness, achieving 92.3% AUROC on viewpoint-independent shape matching tasks, effectively performing a form of "mental rotation."

Crucially, the attributes learned from abstract sculptures generalized to real-world objects. When tested on PASCAL VOC, the model correctly identified the geometric properties of unseen categories—for example, recognizing that trains are typically "planar" and "cylindrical," while cats are "non-planar." This zero-shot generalization allowed for object classification with 89% AUROC based solely on the 12 shape attributes.

Conclusion

This research demonstrates that high-level 3D shape properties can be robustly inferred from single images by training on diverse geometric forms. While the current 12-attribute vocabulary is a foundational step, future work may expand this to include local attribute labeling and relative attributes to better capture the nuances of 3D geometry.

Summary updated

Abstract

Our goal in this paper is to investigate properties of 3D shape that can be determined from a single image. We define 3D shape attributes -- generic properties of the shape that capture curvature, contact and occupied space. Our first objective is to infer these 3D shape attributes from a single image. A second objective is to infer a 3D shape embedding -- a low dimensional vector representing the 3D shape. We study how the 3D shape attributes and embedding can be obtained from a single image by training a Convolutional Neural Network (CNN) for this task. We start with synthetic images so that the contribution of various cues and nuisance parameters can be controlled. We then turn to real images and introduce a large scale image dataset of sculptures containing 143K images covering 2197 works from 242 artists. For the CNN trained on the sculpture dataset we show the following: (i) which regions of the imaged sculpture are used by the CNN to infer the 3D shape attributes; (ii) that the shape embedding can be used to match previously unseen sculptures largely independent of viewpoint; and (iii) that the 3D attributes generalize to images of other (non-sculpture) object classes.
BibTeX
@article{fouhey2016understanding,
  author = {David F. Fouhey and Abhinav Gupta and Andrew Zisserman},
  title = {{Understanding Higher-Order Shape via 3D Shape Attributes}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2016},
  url = {https://openreview.net/forum?id=aBfvHEMTp-M},
}
Figure