Abstract
The goal of correspondence pruning is to recognize correct correspondences (inliers) from initial ones, with applications to various feature matching based tasks. Seeking neighbors in the coordinate and feature spaces is a common strategy in many previous methods. However, it is difficult to ensure that these neighbors are always consistent, since the distribution of false correspondences is extremely irregular. For addressing this problem, we propose a novel global-graph space to search for consistent neighbors based on a weighted global graph that can explicitly explore long-range dependencies among correspondences. On top of that, we progressively construct three neighbor embeddings according to different neighbor search spaces, and design a Neighbor Consistency block to extract neighbor context and explore their interactions sequentially. In the end, we develop a Neighbor Consistency Mining Network (NCMNet) for accurately recovering camera poses and identifying inliers. Experimental results indicate that our NCMNet achieves a significant performance advantage over state-of-the-art competitors on challenging outdoor and indoor matching scenes.
Introduction
In computer vision, aligning two images—a process essential for 3D reconstruction and autonomous navigation—begins with finding "correspondences" or matching points. However, factors like lighting changes, perspective shifts, and repetitive textures often result in a high volume of "outliers" or incorrect matches.
Correspondence pruning is the critical task of filtering these outliers. Traditional methods rely on "neighbor consistency," assuming that if a match is correct, its neighbors in the image should exhibit similar geometric movement. However, because outliers are distributed irregularly, relying solely on local neighborhoods in image coordinates or neural feature spaces often fails to distinguish between true inliers and "consistent-looking" outliers.
The NCMNet Framework
To overcome the limitations of local search, the Neighbor Consistency Mining Network (NCMNet) introduces a global-graph space. This approach builds a weighted global graph where every correspondence is a node, and edges represent the likelihood of two matches being mutual inliers. By employing Spectral Graph Convolution (SGC), the network explores long-range dependencies, identifying globally consistent neighbors even if they are spatially distant in the image.
Progressive Neighbor Consistency (NC) Block
The core of NCMNet is the NC Block, which progressively processes three distinct types of neighbor embeddings:
- Spatial Neighbors: Points close in image coordinates.
- Feature-space Neighbors: Points with similar neural signatures.
- Global-graph Neighbors: Long-range neighbors identified via the global graph.
Within this block, the Self-Context Extraction (SCE) layer uses grouped convolutions to capture relationships within a neighbor group, while the Cross-Context Interaction (CCI) layer utilizes a cross-attention mechanism to fuse information across the different neighbor types. This iterative pruning strategy gradually distills the most reliable matches to estimate the camera's essential matrix accurately.
Experimental Evaluation
The researchers evaluated NCMNet on standard benchmarks including YFCC100M (outdoor) and SUN3D (indoor), using both traditional SIFT features and modern SuperPoint descriptors.
Key Results
- Outdoor Performance: On the YFCC100M dataset, NCMNet achieved an mAP5° of 63.43%, a significant improvement over the previous state-of-the-art, CLNet (53.10%).
- Indoor Performance: On SUN3D, it reached 20.66% mAP5°, outperforming competitors like MS2DG-Net (17.84%).
- Robustness: The model proved remarkably resilient to low inlier ratios. Even when initial matches were 90% outliers, NCMNet effectively identified the correct correspondences.
Conclusion
NCMNet demonstrates that while outliers may appear consistent locally, they rarely maintain consistency when viewed through a global lens. By integrating spatial, feature, and global-graph context, the network provides a robust solution for correspondence pruning. While the computational cost of global graph construction is higher than local methods, the substantial gains in accuracy and generalization across diverse scenes mark a significant advancement in the field of feature matching.
Summary updated