Abstract
Image matching is a fundamental component of various computer vision tasks, yet it remains highly susceptible to performance degradation in the presence of occlusions. Occ2Net is a novel framework that addresses this challenge by incorporating 3D occupancy estimation into the matching pipeline. By modeling the scene's geometry, Occ2Net can infer the locations of points that are visible in one view but obscured in another, significantly enhancing camera pose estimation in complex, real-world environments.
Introduction
Traditional image matching algorithms typically rely on identifying and correlating interest points visible across multiple camera views. However, dynamic or cluttered environments frequently present occlusions, where objects visible in one frame are hidden behind foreground elements in another. Standard methods often fail in these scenarios, leading to fragmented trajectories in SLAM and positioning errors in autonomous systems.
Occ2Net shifts the paradigm from ignoring occluded regions to explicitly modeling them. By leveraging 3D spatial cues, the framework treats occlusions as predictable signals rather than noise, allowing for robust matching even when key features are temporarily obscured.
Methodology
Occ2Net employs a coarse-to-fine matching pipeline consisting of two primary components: the Occlusion-Aware (OA) module and the Occupancy Estimation (OE) module.
Occlusion-Aware (OA) Module
The OA module handles the coarse matching stage by identifying corresponding patches between images. A key innovation within this module is Rotation Alignment. To account for camera rotation, the module locally rotates feature patches to determine the optimal orientation before matching. This inductive bias ensures the model remains robust against significant viewpoint changes.
Occupancy Estimation (OE) Module
The OE module serves as the "fine" stage of the pipeline. Drawing inspiration from Neural Radiance Fields (NeRF), it treats each pixel as a ray and estimates a 3D occupancy grid, represented as a 4D tensor. By integrating these 3D geometric priors with high-resolution image features, the model can accurately infer the coordinates of a point even if it is hidden behind an occluder.
The network is optimized using a multi-task loss function comprising coarse-matching loss, fine-matching (reprojection) loss, and a 3D occupancy loss supervised by ground-truth depth data.
Experimental Results
The performance of Occ2Net was evaluated on two benchmark datasets: ScanNet, which provides noisy real-world indoor data, and TartanAir, a synthetic dataset used to create extreme occlusion scenarios (occlusion ratio > 0.3).
Quantitative Performance
Occ2Net consistently outperformed state-of-the-art (SOTA) baselines, including LoFTR, SuperGlue, and ASpanFormer. On the ScanNet dataset, the Occ2Net-l variant achieved a Pose Estimation AUC@5° of 28.0%, surpassing ASpanFormer (25.6%) and QuadTree LoFTR (24.9%).
The advantage of Occ2Net became even more pronounced in high-occlusion environments. On the challenging TartanAir subset, Occ2Net-l maintained an AUC@5° of 28.0%, while the previous SOTA, QuadTree LoFTR, dropped significantly to 18.3%.
Qualitative Analysis
Visualizations demonstrate that Occ2Net successfully establishes matches on surfaces and objects that are partially or fully obscured by foreground elements. In contrast, competing models often fail to produce matches or generate incorrect correspondences in these regions.
Conclusion
Occ2Net demonstrates that 3D occupancy estimation is a powerful tool for overcoming the limitations of traditional image matching in occluded scenes. By treating occlusions as a geometric property of the environment rather than missing data, the framework achieves superior robustness and accuracy. Future work may explore more complex ray sampling techniques to handle multi-layered occlusions and further refine sub-pixel localization for hidden points.
Summary updated