Introduction
Reconstructing a 3D scene from a single 2D image remains a significant challenge in computer vision. Because a 2D projection loses depth information, the problem is mathematically ill-posed—infinitely many 3D configurations could technically produce the same 2D image.
To address this, researchers often employ the Manhattan World assumption, which posits that most lines in a scene are aligned with three mutually orthogonal axes (e.g., the walls, floors, and ceilings of urban environments). While modern deep learning methods can estimate depth maps, they often lack the geometric precision required for engineering or architectural applications. This paper bridges the gap between classical geometric reasoning and modern computer vision by enforcing "physical realizability" on 3D line reconstructions.
Methodology
The authors model the reconstruction as a "straws and connectors" problem. Detected line segments are represented as vertices in a graph, where potential intersections are edges. The primary task is to distinguish between "true" intersections (where lines actually touch in 3D space) and "fake" intersections (where lines merely overlap from the camera's perspective).
The core of the approach is a Mixed Integer Linear Program (MILP) that incorporates three novel sets of constraints:
- Cycle Graph Constraints: In 3D space, a loop of segments (e.g., moving along X, then Y, then Z axes) must return to the starting point to close. The authors identify specific 2D cycles that are physically impossible in 3D and use constraints to prevent the optimizer from selecting them.
- Planarity Constraints: If two parallel lines lie on the same Manhattan plane, it dictates the possible interactions of other lines with that plane.
- Boundary Constraints: Lines are classified as either "boundaries" (where two planes meet) or "non-boundaries" (lines on a flat surface), restricting how they can intersect lines from other directions.
Unlike previous methods using L1-norm relaxation, this MILP formulation provides an exact solution to the connectivity problem, ensuring a valid, connected graph.
Key Insights: The Geometry of Cycles
A pivotal insight of this work is that 2D geometry can be deceptive. For instance, a triangle formed by an X-line, a Y-line, and a Z-line in a 2D image cannot form a closed triangle in 3D space.
By explicitly forbidding these impossible 3D structures, the search space for the reconstruction is dramatically pruned. This highlights the value of hard geometric rules over "black-box" neural networks for tasks requiring high precision; fundamental laws of physics do not need to be learned if they can be mathematically enforced.
Datasets and Results
The authors introduced a new Single View Line Reconstruction dataset consisting of 50 high-resolution (4K) photos of urban scenes captured via a car-mounted GoPro. This dataset provides significantly larger depth variations than standard benchmarks like the York Urban database.
An ablation study demonstrated that each constraint type improves accuracy over existing baselines:
- Baseline Accuracy: 80.58%
- With Boundary Constraints: 82.89%
- With All Constraints (B + C + P): 83.21%
Qualitatively, the system produces precise 3D wireframes from single street-view images. The optimization is efficient, averaging approximately 18 seconds per image on a standard desktop.
Limitations and Future Work
Currently, the algorithm only reconstructs the largest connected component of the line graph, meaning disconnected structures are lost. Furthermore, the method relies heavily on the orthogonal Manhattan assumption and may struggle with organic or curved architecture. Future research may incorporate occluding contours to move toward full plane-based 3D models.
Summary updated