Introduction
Estimating the 3D structure of an object from 2D images is a cornerstone of computer vision. While traditional Structure from Motion (SfM) typically requires multiple views of the exact same rigid object, humans can easily infer 3D geometry from a single image or by observing different instances of a category (e.g., various types of cars). This paper bridges this gap by leveraging geometric priors common in man-made objects: bilateral symmetry and Manhattan structures.
Problem Statement
3D reconstruction from a single view is inherently ill-posed, as infinite 3D configurations can project to the same 2D points. To resolve this ambiguity, the authors utilize two primary constraints:
- Bilateral Symmetry: Most man-made objects, such as cars and planes, exhibit left-right mirror symmetry.
- Manhattan Structure: Many objects possess three identifiable perpendicular axes representing length, width, and height.
Methodology
The authors propose two distinct approaches based on an orthographic projection model.
Single Image Reconstruction
If three Manhattan axes can be identified in a single image, the camera's projection matrix (orientation) can be recovered. Combined with bilateral symmetry constraints, this allows for a unique solution of the 3D coordinates of the object's keypoints. This approach effectively decouples pose estimation from shape estimation.
Symmetric Rigid Structure from Motion (Sym-RSfM)
When Manhattan axes are occluded, the authors extend the framework to multiple images of different objects within the same category. They formulate a "Full Energy" function for Maximum Likelihood Estimation that incorporates symmetry constraints directly into the SfM pipeline. To avoid local minima, a "surrogate energy" function is solved via Singular Value Decomposition (SVD) to provide a robust initialization. The method also iteratively handles occluded keypoints during reconstruction by estimating missing data points.
Key Results
The proposed Sym-RSfM method was evaluated on the Pascal3D+ dataset, covering categories like aeroplanes, cars, buses, and sofas.
Sym-RSfM consistently outperformed standard RSfM and Column Space Fitting (CSF) baselines. In the "aeroplane" category, mean rotation error was reduced from 0.52 (standard RSfM) to 0.13. The method also demonstrated significant robustness to noise in 2D keypoint annotations, benefiting from the stabilizing influence of symmetry priors. Even with limited data, the single-view Manhattan+Symmetry approach produced results comparable to multi-image methods lacking these priors.
Conclusion
By formalizing symmetry and Manhattan properties, this work demonstrates that accurate 3D reconstruction is possible from sparse views and across different object instances. Future work includes moving beyond the orthographic assumption to handle perspective distortion and integrating automated keypoint detection via deep learning to replace manual annotations.
Summary updated