Abstract
Reconstructing detailed 3D scenes from a single 2D image remains a fundamental challenge in computer vision due to inherent depth ambiguity. This paper presents a stochastic grammar framework that integrates geometric constraints—such as orthogonality and co-planarity—with a Hybrid Monte Carlo (HMC) sampling engine. By treating reconstruction as a joint task of image segmentation and geometric optimization, the proposed method recovers fine-grained structures like window recesses and road markings that are typically lost in traditional "blobby" reconstructions. Evaluated on benchmarks including the newly introduced LMW-C dataset, the approach demonstrates significant improvements in surface orientation accuracy and qualitative depth map clarity.
Introduction
Single-view 3D reconstruction is an ill-posed problem because a single 2D projection can correspond to infinitely many 3D configurations. Traditional methods often struggle with two primary issues: the loss of fine-grained details, resulting in overly simplified or "blobby" models, and geometric ambiguity caused by perspective effects. For instance, two lines appearing orthogonal in a 2D image may not be so in 3D space.
The authors propose that these fine-grained details are governed by latent geometric relationships, such as parallelism and co-planarity. The core challenge lies in mathematically enforcing these rules while simultaneously segmenting the image and estimating 3D coordinates.
Methodology: Stochastic Image Grammar
The proposed framework treats 3D reconstruction as a joint optimization problem. It utilizes a Stochastic Image Grammar to describe how image entities (edges and planar regions) relate to one another in 3D space.
The Grammar Rules
The grammar consists of five core rules ($R_1$ to $R_5$) that define the structural hierarchy of the scene:
- $R_1$ & $R_2$: Govern edge relationships, specifically co-linearity and orthogonality.
- $R_3$: Links an edge to a plane, ensuring the line lies precisely on the surface.
- $R_4$ & $R_5$: Manage relationships between neighboring planes, determining if they belong to the same surface or meet at right angles (e.g., a wall-floor junction).
Inference via Hybrid Monte Carlo (HMC)
To navigate the complex solution space, the authors employ a Hybrid Monte Carlo (HMC) sampling algorithm. Unlike standard Markov Chain Monte Carlo (MCMC) methods that rely on slow random walks, HMC utilizes:
- Hamiltonian Dynamics: Leverages gradients of the error function to make large, informed steps in continuous 3D space (depth and orientation).
- Cluster Dynamics: Enables rapid partitioning of the image by flipping labels for entire pixel groups simultaneously.
Datasets and Evaluation
The model was evaluated across four datasets: the CMU benchmark, LMW-A, LMW-B, and a new dataset introduced by the authors, LMW-C. LMW-C consists of 950 images from PASCAL VOC and LabelMe, manually annotated with vanishing points, region labels, and surface normals to focus on "Local Manhattan World" scenes.
Key Results
The complete model, Ours-III, significantly outperformed state-of-the-art baselines in both quantitative and qualitative metrics.
- Surface Orientation: On the CMU dataset, the model achieved 82.31% accuracy, a marked improvement over the 73.72% achieved by previous methods.
- Region Labeling: The model demonstrated a nearly 20% improvement in "best spatial support" scores on the LMW-C dataset.
- Qualitative Sharpness: Visually, the recovered depth maps are significantly sharper than those produced by older geometric context methods, successfully capturing recessed windows and crisp architectural corners.
Discussion and Limitations
A key insight of this work is that joint optimization—performing segmentation and 3D estimation simultaneously—allows geometric rules to improve segmentation accuracy. The use of Hamiltonian dynamics allows the algorithm to follow the "mathematical gravity" of the scene's geometry, exploring the solution space more efficiently than traditional wiggling of pixel depths.
However, the method relies heavily on the Manhattan World Assumption, making it highly effective for urban environments but potentially less robust for organic, curved scenes. Additionally, the computational cost of approximately 50 seconds per image currently limits its use in real-time applications like autonomous driving. Future work may extend this grammar-based framework to temporal domains, such as joint object tracking and segmentation in video.
Summary updated