Exploiting Symmetry and/or Manhattan Properties for 3D Object Structure Estimation from Single and Multiple Images

Yuan Gao, Alan L. Yuille

Introduction

Estimating the 3D structure of an object from 2D images is a cornerstone of computer vision. While traditional Structure from Motion (SfM) typically requires multiple views of the exact same rigid object, humans can easily infer 3D geometry from a single image or by observing different instances of a category (e.g., various types of cars). This paper bridges this gap by leveraging geometric priors common in man-made objects: bilateral symmetry and Manhattan structures.

Problem Statement

3D reconstruction from a single view is inherently ill-posed, as infinite 3D configurations can project to the same 2D points. To resolve this ambiguity, the authors utilize two primary constraints:

  1. Bilateral Symmetry: Most man-made objects, such as cars and planes, exhibit left-right mirror symmetry.
  2. Manhattan Structure: Many objects possess three identifiable perpendicular axes representing length, width, and height.

Methodology

The authors propose two distinct approaches based on an orthographic projection model.

Single Image Reconstruction

If three Manhattan axes can be identified in a single image, the camera's projection matrix (orientation) can be recovered. Combined with bilateral symmetry constraints, this allows for a unique solution of the 3D coordinates of the object's keypoints. This approach effectively decouples pose estimation from shape estimation.

Illustration of Manhattan axes (three perpendicular directions) on a car model used to recover camera orientation.

Symmetric Rigid Structure from Motion (Sym-RSfM)

When Manhattan axes are occluded, the authors extend the framework to multiple images of different objects within the same category. They formulate a "Full Energy" function for Maximum Likelihood Estimation that incorporates symmetry constraints directly into the SfM pipeline. To avoid local minima, a "surrogate energy" function is solved via Singular Value Decomposition (SVD) to provide a robust initialization. The method also iteratively handles occluded keypoints during reconstruction by estimating missing data points.

Key Results

The proposed Sym-RSfM method was evaluated on the Pascal3D+ dataset, covering categories like aeroplanes, cars, buses, and sofas.

Qualitative results of 3D reconstruction for various aeroplane instances, showing input images, 2D keypoints, and the resulting 3D structures.

Sym-RSfM consistently outperformed standard RSfM and Column Space Fitting (CSF) baselines. In the "aeroplane" category, mean rotation error was reduced from 0.52 (standard RSfM) to 0.13. The method also demonstrated significant robustness to noise in 2D keypoint annotations, benefiting from the stabilizing influence of symmetry priors. Even with limited data, the single-view Manhattan+Symmetry approach produced results comparable to multi-image methods lacking these priors.

Conclusion

By formalizing symmetry and Manhattan properties, this work demonstrates that accurate 3D reconstruction is possible from sparse views and across different object instances. Future work includes moving beyond the orthographic assumption to handle perspective distortion and integrating automated keypoint detection via deep learning to replace manual annotations.

Summary updated

BibTeX
@inproceedings{gao2017exploiting,
  author = {Yuan Gao and Alan L. Yuille},
  title = {{Exploiting Symmetry and/or Manhattan Properties for 3D Object Structure Estimation from Single and Multiple Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2017},
  url = {https://openreview.net/forum?id=BJ41cR-dbB},
}
Figure