TL;DR
This paper presents a practical method for creating 3D building models from uncalibrated video sequences by integrating scene-specific geometric knowledge (parallel lines, orthogonal planes) into the auto-calibration stage of Structure from Motion (SFM). This approach improves the robustness and Euclidean accuracy of reconstructions, especially in challenging "critical motion" scenarios where standard methods fail.
What Problem Does This Paper Address?
Reconstructing accurate 3D models of real-world objects, like buildings, from uncalibrated video is a long-standing challenge in computer vision. The "Structure from Motion" (SFM) problem aims to recover both the 3D structure of a scene and the camera poses that captured it. A significant hurdle is "uncalibrated SFM," where the camera's internal parameters (like focal length and lens distortion) are unknown. This often requires a two-stage process: first, a projective reconstruction, and then "auto-calibration" to convert it to a more geometrically accurate Euclidean reconstruction. However, auto-calibration can fail or become ambiguous when the camera motion is "critical" (e.g., pure rotation or translation along a line). This paper tackles the problem of making auto-calibration more robust and accurate, particularly for building reconstruction, by leveraging inherent geometric properties of buildings themselves.
Methods
The core novelty of this work lies in incorporating scene-specific geometric constraints into the auto-calibration process.
- Auto-calibration: The paper builds upon existing auto-calibration techniques that aim to determine the "absolute quadric" ($Q_\infty$), which encodes Euclidean structure. Standard methods rely on camera-based constraints derived from the properties of $Q_\infty$ under Euclidean transformations.
- Scene-based Constraints: The authors introduce two key types of scene-based constraints derived from the geometric properties of buildings:
- Parallel Lines: Buildings often have parallel lines (e.g., edges of windows, roof lines). The direction points (points at infinity) of these parallel lines must lie on the plane at infinity. This property is used to constrain the absolute quadric.
- Orthogonal Planes: Buildings also feature orthogonal planes (e.g., walls meeting at a corner). The paper shows that for two orthogonal planes, their normal directions are orthogonal with respect to the absolute quadric. This translates into linear constraints on the entries of $Q_\infty$.
- Integration: These scene-based constraints are integrated with the traditional camera-based constraints to form a unified system for estimating the absolute quadric. This combined approach is designed to resolve ambiguities in critical motion cases and improve the overall accuracy and robustness of the Euclidean reconstruction. An algorithm is proposed to obtain an initial estimate of $Q_\infty$ using these constraints, followed by a non-linear refinement.
Datasets
The paper evaluates its approach using two types of data:
- Simulated Data: A synthetic scene with known geometric properties (three pairs of orthogonal planes and three sets of parallel lines) was generated. This allowed for controlled experiments with varying numbers of views (4 and 10) and added noise levels (1 and 2 pixels) to assess robustness.
- Real Image Data: Two real image sequences of buildings were used. The paper shows reconstructed models from these sequences, both with and without the proposed scene-based constraints.
Key Insights and Ideas
- Leveraging Building Geometry: The most significant insight is that the inherent geometric properties of buildings (parallelism and orthogonality) are not just visual cues but can be mathematically formulated as powerful constraints for solving challenging computer vision problems like auto-calibration.
- Resolving Critical Motion Ambiguities: Standard auto-calibration methods struggle with "critical motion" sequences. By incorporating scene-based constraints, the paper demonstrates a way to overcome these ambiguities and achieve a more reliable Euclidean reconstruction even in these difficult scenarios.
- Improved Euclidean Accuracy: The scene-based constraints directly enforce Euclidean properties, leading to reconstructions that are visually more accurate and less prone to projective distortions, especially when camera motion is limited or repetitive.
- Unified Constraint Framework: The paper successfully integrates camera-based and scene-based constraints into a single framework, showing that combining different sources of information can lead to more robust solutions.
Key Results
- Simulated Data: Experiments on simulated data showed that the proposed method (Method B) significantly reduced auto-calibration residual errors and improved the accuracy of reconstructed orthogonal planes and parallel line intersections compared to a method without scene-based constraints (Method A). This improvement was observed across different view numbers and noise levels.
- Real Image Data: For the first real image sequence, the improvements were less pronounced, suggesting that standard methods can already yield good results when camera motion is sufficiently general. However, for the second sequence, which exhibited projective distortions even with small auto-calibration errors, the proposed method produced a noticeably more Euclidean and visually pleasing reconstruction.
Limitations and Future Work
- Requirement for Scene Knowledge: The approach relies on the presence of parallel lines and orthogonal planes within the scene. If a building lacks these features or they are not clearly visible, the method's effectiveness will be diminished.
- Manual Extraction of Constraints: While the paper describes how to extract these constraints, the process might still involve some level of manual intervention or robust feature detection, which could be a bottleneck for fully automatic systems.
- Future Work: The authors suggest that further integration of other scene properties could lead to even more robust and accurate reconstructions. The paper also implies that extending this approach to more complex scenes or different types of objects could be a valuable direction.
Summary updated