Occ2Net: Robust Image Matching Based on 3D Occupancy Estimation for Occluded Regions.

Miao Fan, Mingrui Chen, Chen Hu, Shuchang Zhou

Abstract

Image matching is a fundamental component of various computer vision tasks, yet it remains highly susceptible to performance degradation in the presence of occlusions. Occ2Net is a novel framework that addresses this challenge by incorporating 3D occupancy estimation into the matching pipeline. By modeling the scene's geometry, Occ2Net can infer the locations of points that are visible in one view but obscured in another, significantly enhancing camera pose estimation in complex, real-world environments.

Introduction

Traditional image matching algorithms typically rely on identifying and correlating interest points visible across multiple camera views. However, dynamic or cluttered environments frequently present occlusions, where objects visible in one frame are hidden behind foreground elements in another. Standard methods often fail in these scenarios, leading to fragmented trajectories in SLAM and positioning errors in autonomous systems.

Conceptual overview of matching occluded points using 3D occupancy.

Occ2Net shifts the paradigm from ignoring occluded regions to explicitly modeling them. By leveraging 3D spatial cues, the framework treats occlusions as predictable signals rather than noise, allowing for robust matching even when key features are temporarily obscured.

Methodology

Occ2Net employs a coarse-to-fine matching pipeline consisting of two primary components: the Occlusion-Aware (OA) module and the Occupancy Estimation (OE) module.

The Occ2Net architecture featuring OA and OE modules.

Occlusion-Aware (OA) Module

The OA module handles the coarse matching stage by identifying corresponding patches between images. A key innovation within this module is Rotation Alignment. To account for camera rotation, the module locally rotates feature patches to determine the optimal orientation before matching. This inductive bias ensures the model remains robust against significant viewpoint changes.

Occupancy Estimation (OE) Module

The OE module serves as the "fine" stage of the pipeline. Drawing inspiration from Neural Radiance Fields (NeRF), it treats each pixel as a ray and estimates a 3D occupancy grid, represented as a 4D tensor. By integrating these 3D geometric priors with high-resolution image features, the model can accurately infer the coordinates of a point even if it is hidden behind an occluder.

The network is optimized using a multi-task loss function comprising coarse-matching loss, fine-matching (reprojection) loss, and a 3D occupancy loss supervised by ground-truth depth data.

Experimental Results

The performance of Occ2Net was evaluated on two benchmark datasets: ScanNet, which provides noisy real-world indoor data, and TartanAir, a synthetic dataset used to create extreme occlusion scenarios (occlusion ratio > 0.3).

Quantitative Performance

Occ2Net consistently outperformed state-of-the-art (SOTA) baselines, including LoFTR, SuperGlue, and ASpanFormer. On the ScanNet dataset, the Occ2Net-l variant achieved a Pose Estimation AUC@5° of 28.0%, surpassing ASpanFormer (25.6%) and QuadTree LoFTR (24.9%).

The advantage of Occ2Net became even more pronounced in high-occlusion environments. On the challenging TartanAir subset, Occ2Net-l maintained an AUC@5° of 28.0%, while the previous SOTA, QuadTree LoFTR, dropped significantly to 18.3%.

Qualitative Analysis

Visualizations demonstrate that Occ2Net successfully establishes matches on surfaces and objects that are partially or fully obscured by foreground elements. In contrast, competing models often fail to produce matches or generate incorrect correspondences in these regions.

Conclusion

Occ2Net demonstrates that 3D occupancy estimation is a powerful tool for overcoming the limitations of traditional image matching in occluded scenes. By treating occlusions as a geometric property of the environment rather than missing data, the framework achieves superior robustness and accuracy. Future work may explore more complex ray sampling techniques to handle multi-layered occlusions and further refine sub-pixel localization for hidden points.

Summary updated

Abstract

Image matching is a fundamental and critical task in various visual applications, such as Simultaneous Localization and Mapping (SLAM) and image retrieval, which require accurate pose estimation. However, most existing methods ignore the occlusion relations between objects caused by camera motion and scene structure. In this paper, we propose Occ2Net, a novel image matching method that models occlusion relations using 3D occupancy and infers matching points in occluded regions. Thanks to the inductive bias encoded in the Occupancy Estimation (OE) module, it greatly simplifies bootstrapping of a multi-view consistent 3D representation that can then integrate information from multiple views. Together with an Occlusion-Aware (OA) module, it incorporates attention layers and rotation alignment to enable matching between occluded and visible points. We evaluate our method on both real-world and simulated datasets and demonstrate its superior performance over state-of-the-art methods on several metrics, especially in occlusion scenarios.
BibTeX
@inproceedings{fan2023occ,
  author = {Miao Fan and Mingrui Chen and Chen Hu and Shuchang Zhou},
  title = {{Occ2Net: Robust Image Matching Based on 3D Occupancy Estimation for Occluded Regions.}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=RO9qkw8vNG},
}
Figure