Abstract
What primitives should we use to infer the rich 3D world behind an image? We argue that these primitives should be both visually discriminative and geometrically informative and we present a technique for discovering such primitives. We demonstrate the utility of our primitives by using them to infer 3D surface normals given a single image. Our technique substantially outperforms the state-of-the-art and shows improved cross-dataset performance.
Introduction
Inferring the 3D structure of a scene from a single 2D image is a fundamental challenge in computer vision. Historically, research has oscillated between two extremes: using simple geometric primitives like "geons" (cylinders, spheres), which are geometrically meaningful but difficult to detect in real-world scenes, and using appearance-based local features, which are easy to detect but often lack a consistent relationship to the underlying 3D structure.
This paper bridges the gap by introducing a data-driven framework to discover a vocabulary of "3D primitives"—mid-level visual patches that are both easy to detect in 2D images and carry consistent 3D geometric information. By learning these primitives from RGB-D data, the authors achieve state-of-the-art results in estimating dense 3D surface normals from a single RGB image.
The 3D Primitive Framework
The proposed framework discovers primitives that balance visual discriminability with geometric informativeness. Each primitive in the learned vocabulary consists of three components: an appearance model (a discriminative linear detector using HOG features), a geometric "canonical form" (the average 3D surface normals for that patch), and specific training instances.
Learning and Inference
Because the optimal patches are not known a priori, the authors use an iterative optimization approach. Patches are initially grouped by similarity in both appearance and geometry. An SVM detector is then trained to distinguish these groups from the background. Through a refinement process, the system identifies more examples, updates the canonical 3D shape, and retrains the detectors using a cross-validation strategy to prevent overfitting.
To process a new image, the system runs thousands of primitive detectors. Because these detections are sparse, the final surface normal map is generated via a weighted linear sum (label transfer). Detections with higher confidence or closer spatial proximity to a pixel exert a greater influence on that pixel's predicted normal.
Key Insights
- Visual Synonyms: The model discovers that disparate objects can share identical 3D geometry. For instance, the corner of a wooden cabinet and the corner of a white wall may be "visual synonyms" representing the same 3D intersection of planes.
- Discriminative + Informative: The objective function explicitly balances the ease of detection with the consistency of the underlying 3D shape. A primitive is only useful if it can be reliably found and provides a stable geometric signal.
- Beyond Manhattan Constraints: Unlike many 3D solvers that assume a "Manhattan World" (where all surfaces are axis-aligned), these primitives capture complex local geometries without requiring global scene consistency.
Experimental Results
The method was primarily evaluated on the NYU Depth v2 dataset (1,449 indoor RGB-D images). It significantly outperformed existing baselines:
- Surface Normal Accuracy: The proposed method achieved a Mean Error of 33.5° (with Manhattan constraints), compared to 41.2° for the leading baseline at the time (Hedau et al.).
- Robustness: The improvement in Median Error was even more significant (18.0° vs. 25.5°), indicating a more robust model.
- Generalization: When trained on NYU and tested on the Berkeley 3D Object Dataset (B3DO), the model maintained its performance lead, suggesting that the learned primitives represent fundamental properties of indoor environments rather than dataset-specific artifacts.
Conclusion and Future Work
While the method excels at capturing local geometry, it faces challenges in large, textureless regions where visual detectors lack sufficient evidence. Future work may involve integrating global scene context, such as overall room layout, to resolve local ambiguities and improve performance in feature-poor areas of an image.
Summary updated