Fixed Viewpoint Mirror Surface Reconstruction under an Uncalibrated Camera

Kai Han, Miaomiao Liu, Dirk Schnieders, Kwan-Yee K. Wong

Abstract

Reconstructing mirror surfaces is a significant challenge in computer vision due to their viewpoint-dependent appearance. This paper proposes a method to reconstruct 3D mirror surfaces using a single stationary, uncalibrated camera and a moving reference plane. By observing reflections of the plane in at least three unknown poses, the method recovers camera intrinsics, plane poses, and the mirror's 3D shape. The approach utilizes reflection correspondences and a cross-ratio based optimization to achieve high precision without tedious prior calibration.

Introduction

While 3D reconstruction of matte objects has reached high levels of maturity, specular surfaces like mirrors remain problematic. Standard techniques like Structure-from-Motion rely on the "brightness constancy constraint," which assumes a point's appearance is invariant to the viewing angle. Mirrors violate this, as their appearance is a reflection of the surrounding environment.

To address this, "Shape from Specular Reflection" techniques typically use a screen to display patterns reflected off the mirror. However, most existing methods require a fully calibrated setup, including precise knowledge of camera intrinsics and the exact 3D motion of the reference screen. This paper introduces a novel pipeline that removes these requirements, enabling mirror reconstruction in uncalibrated, "in-the-wild" settings.

Methodology

The proposed reconstruction pipeline leverages the geometric properties of light paths and projective invariants.

Reflection Geometry and the "Power of Three"

The core idea is to establish reflection correspondences between 3D points on an LCD screen and 2D pixels in the camera. By moving the screen to at least three unknown positions, the system gains enough geometric constraints to solve for the relative 3D motion of the screen and the camera parameters simultaneously.

Figure 3: Geometric model of the mirror surface reconstruction using three poses of a reference plane.

As shown in the geometric model, light rays from points $X_0, X_1, X_2$ on planes $P_0, P_1, P_2$ reflect off the mirror surface $S$ at point $M$ and are captured at image point $m$ by camera $C$. The authors represent these light paths as 3D lines using Plücker coordinates, deriving a "line projection matrix" that can be converted into a standard camera projection matrix.

Cross-Ratio Optimization

To ensure robustness against noise and lens distortion, the method employs the cross-ratio—a fundamental invariant in projective geometry. The camera projection matrix is optimized by minimizing reprojection errors based on this cross-ratio constraint. This approach is significantly more stable than traditional point-to-line distance metrics, which often suffer from "sliding" errors in 3D space.

Experimental Evaluation

The method was validated through both synthetic simulations and real-world experiments.

Setup and Data

For real-world testing, the researchers used a 19-inch LCD monitor as the reference plane and a Canon EOS 40D digital camera. Objects such as a metallic sauce boat and chrome spheres were reconstructed.

Figure 1: Experimental setup showing the camera, the reference LCD screen with a checkerboard pattern, and the reflected images.

Synthetic tests utilized standard models like the "Stanford Bunny" and an "engine hood" to evaluate performance against ground truth under varying noise levels.

Key Results

  • Precision: Synthetic tests showed camera intrinsic errors (e.g., focal length) below 1%, with rotation errors as low as 0.08 degrees.
  • Accuracy: Real-world reconstructions of objects (20-30cm in size) achieved a Root Mean Square (RMS) error consistently below 3mm.
  • Robustness: The cross-ratio optimization effectively handled lens distortion, producing accurate 3D point clouds even without explicit initial radial distortion modeling.

Conclusion and Future Work

This research demonstrates that mirror surfaces can be reconstructed with high accuracy using uncalibrated equipment. By treating the problem as a light path triangulation, the mirror's viewpoint dependency becomes a geometric advantage.

However, limitations remain. The mathematical model can fail for "degenerate" shapes like perfect planes or specific elliptical curves. Additionally, the current method assumes single reflections, making it unsuitable for complex concave surfaces where inter-reflections occur. Future work may address these complexities and extend the method to non-perfectly specular (glossy) materials.

Summary updated

Abstract

This paper addresses the problem of mirror surface reconstruction, and proposes a solution based on observing the reflections of a moving reference plane on the mirror surface. Unlike previous approaches which require tedious calibration, our method can recover the camera intrinsics, the poses of the reference plane, as well as the mirror surface from the observed reflections of the reference plane under at least three unknown distinct poses. We first show that the 3D poses of the reference plane can be estimated from the reflection correspondences established between the images and the reference plane. We then form a bunch of 3D lines from the reflection correspondences, and derive an analytical solution to recover the line projection matrix. We transform the line projection matrix to its equivalent camera projection matrix, and propose a cross-ratio based formulation to optimize the camera projection matrix by minimizing reprojection errors. The mirror surface is then reconstructed based on the optimized cross-ratio constraint. Experimental results on both synthetic and real data are presented, which demonstrate the feasibility and accuracy of our method.
BibTeX
@article{han2021fixed,
  author = {Kai Han and Miaomiao Liu and Dirk Schnieders and Kwan-Yee K. Wong},
  title = {{Fixed Viewpoint Mirror Surface Reconstruction under an Uncalibrated Camera}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=6fj1z3n4zPq},
}
Figure