Abstract
Traditional RGB-D reconstruction methods often fail in featureless indoor environments or when camera loops have minimal overlap. PlaneMatch addresses these challenges by introducing a deep-learning-based descriptor that identifies coplanar surfaces across different views ab initio. By integrating these long-range geometric constraints into a global optimization framework, PlaneMatch achieves robust, high-precision 3D reconstruction where traditional keypoint-based methods struggle.
Introduction
Standard SLAM and 3D reconstruction pipelines rely heavily on frame-to-frame alignment (e.g., ICP) and local keypoint matching (e.g., SIFT). However, indoor scenes frequently contain large, textureless surfaces like white walls or floors that lack sufficient unique points for tracking. Furthermore, significant drift can occur when scanning paths do not frequently overlap, preventing "loop closure."
PlaneMatch moves beyond local point matching by leveraging the inherent structure of indoor environments: planes. While humans can intuitively recognize that separate patches of a floor belong to the same infinite plane, computers typically require prior alignment to make such a determination. PlaneMatch provides a descriptor capable of predicting coplanarity without any initial pose estimate.
Methods: The Coplanarity Network
The core innovation is a Siamese-style deep CNN designed to generate a 256-dimensional descriptor for planar patches.
Multi-Scale Architecture
The network employs two "towers" to capture both local detail and global context:
- Local Tower: Processes the patch and its immediate neighborhood (1.5x bounding box).
- Global Tower: Captures a 5x bounding box to provide structural cues, enabling the network to "reason" about the room's layout.
Input and Masking
The network accepts a multi-channel stack including RGB, Depth, Surface Normals, and a Mask. A "global decay" mask is used in the global tower, smoothly fading away from the target patch to maintain focus while preserving surrounding context.
Training and Optimization
The model was trained on 10 million triplets from the ScanNet dataset using a Triplet Focal Loss. This loss function prioritizes "hard" ambiguous cases, preventing the network from being dominated by easy negative examples. For reconstruction, these descriptors are integrated into a pose-graph optimization that minimizes point-to-plane distances, using indicator variables to prune false matches automatically.
Key Insights
- Coplanarity as Semantic Reasoning: The authors demonstrate that predicting coplanarity is a visual reasoning task. Understanding the room's structure via the global context tower is essential for linking distant surfaces.
- Geometric Dominance: Ablation studies reveal that surface normals and masks are more discriminative than raw RGB or depth data, highlighting the importance of geometric shape over color.
- Long-Range Constraints: Unlike traditional methods requiring physical overlap, PlaneMatch can link surfaces across different rooms (e.g., the same floor plane), effectively mitigating global drift.
Key Results
PlaneMatch was evaluated on the TUM RGB-D and Sun3D benchmarks, as well as a new COP Benchmark (comprising 12,000 patch pairs).
- Accuracy: On the TUM benchmark, PlaneMatch achieved an Absolute Trajectory Error (ATE) as low as 1.1 cm, rivaling or exceeding state-of-the-art systems like BundleFusion.
- Robustness: The optimization framework remains effective even with a precision as low as 20% (80% incorrect matches), provided recall is maintained.
- Low-Overlap Success: Qualitative results demonstrate successful reconstruction in scenarios where keypoint-based methods fail due to insufficient frame overlap.
Limitations and Future Work
While robust, the system faces challenges in under-constrained scenes (e.g., viewing a single flat wall) where coplanarity alone cannot resolve all degrees of freedom. Additionally, the global optimization process currently takes approximately 20 minutes, making it unsuitable for real-time applications. Future work may focus on accelerating this optimization and improving disambiguation in highly repetitive featureless environments.
Summary updated