Abstract
This paper presents a unified grammatical framework capable of reconstructing a variety of scene types (e.g., urban, campus, county etc.) from a single input image. The key idea of our approach is to study a novel commonsense reasoning framework that mainly exploits two types of prior knowledges: (i) prior distributions over a single dimension of objects, e.g., that the length of a sedan is about 4.5 meters; (ii) pair-wise relationships between the dimensions of scene entities, e.g., that the length of a sedan is shorter than a bus. These unary or relative geometric knowledge, once extracted, are fairly stable across different types of natural scenes, and are informative for enhancing the understanding of various scenes in both 2D images and 3D world. Methodologically, we propose to construct a hierarchical graph representation as a unified representation of the input image and related geometric knowledge. We formulate these objectives with a unified probabilistic formula and develop a data-driven Monte Carlo method to infer the optimal solution with both bottom-to-up and top-down computations. Results with comparisons on public datasets showed that our method clearly outperforms the alternative methods.
Introduction
Reconstructing a 3D scene from a single 2D image is a fundamental challenge in computer vision. It is an "ill-posed" problem because infinitely many 3D configurations can project onto the same 2D image. While humans navigate this ambiguity by relying on innate spatial reasoning, artificial systems often struggle. Most existing algorithms are "specialists" designed for specific environments, such as Manhattan-style urban centers with rigid parallel lines. However, these models often fail in "messier" real-world environments like rural roads or suburban campuses.
This research introduces a framework that leverages "geometric commonsense"—universal prior knowledge about object sizes and their relationships—to parse and reconstruct diverse 3D scenes from a single image.
Methodology
The core of the proposed approach is a Stochastic Attribute Scene Grammar that bridges the gap between 2D pixels and 3D space.
The Parse Graph
The system represents an image as a hierarchical parse graph. The root node represents the entire "Scene," which decomposes into background elements (roads, buildings) and foreground objects (humans, vehicles). Each node contains attributes such as 3D position, orientation, and physical dimensions.
Geometric Commonsense Priors
The framework formalizes two types of prior knowledge:
- Type-I (Unary): Absolute distributions of dimensions. For instance, the model assumes a male adult is approximately 1.77m tall and a sedan is roughly 4.6m long.
- Type-II (Pairwise): Relative relationships between entities. This includes 21 rules, such as "windows on a facade are parallel" and "a building is taller than a human."
Inference via DDMCMC
To find the optimal 3D interpretation, the authors employ Data-Driven Markov Chain Monte Carlo (DDMCMC). This process involves a bidirectional computation:
- Bottom-up: Detecting low-level features like edges, superpixels, and objects.
- Top-down: Using the grammar and commonsense rules to propose 3D model adjustments, ensuring the reconstruction is physically plausible and consistent with 2D evidence.
Experimental Results
The authors curated a new dataset of 2,500 images across five categories (Country, Suburban, Road, Campus, and Urban) to test the model's generalization capabilities. The proposed method was compared against high-profile baselines from Hoiem et al., Gupta et al., and Liu et al.
- Surface Orientation: The method significantly outperformed baselines across all categories. In "Country" scenes, it achieved 64.4% accuracy, compared to 54.6% for the next best baseline.
- Segmentation: The model showed substantial improvement in region segmentation, outperforming Liu et al. by 11.4% in country scenes.
- Qualitative Depth: As shown in the reconstruction results, the depth maps capture fine-grained details like small facades and windows that baseline methods typically smooth over.
Conclusion
This framework demonstrates that geometric commonsense is a powerful, universal anchor for 3D reasoning. By jointly optimizing for object labels and depth, the model achieves state-of-the-art performance across diverse environments. While the current inference speed (1–2 minutes per image) and reliance on initial 2D detectors are limitations, the approach provides a robust foundation for future work in general-purpose 3D scene understanding.
Summary updated