DeMo: Deep Motion Field Consensus with Learnable Kernels for Two-view Correspondence Learning

Yifan Lu, Jiajun Le, Zizhuo Li, Yixuan Yuan, Jiayi MaAAAI 2025 · DOI

TL;DR

This paper introduces DeMo, a novel deep learning framework for robustly finding corresponding points between two images. DeMo leverages "motion consensus" – the idea that the overall movement between images should be smooth and consistent – by using learnable kernels to capture global motion patterns. This approach significantly improves accuracy in tasks like 3D reconstruction and visual localization, especially in challenging scenarios with many incorrect initial matches.

What Problem Does This Paper Address?

Accurately identifying corresponding points (keypoints) between two images of the same scene is a fundamental problem in computer vision, crucial for applications like 3D reconstruction, augmented reality, and visual localization. The standard pipeline involves detecting keypoints, describing them with features, and then matching these features to establish correspondences. However, these initial correspondences are often noisy, containing many mismatches (outliers), especially in complex scenes with viewpoint changes or illumination variations. Rejecting these outliers to obtain a clean set of correspondences is critical for accurate downstream tasks. While existing methods have made progress, they often struggle to effectively capture the global consistency of motion across the entire image pair, particularly when dealing with significant depth discontinuities or a high percentage of outliers.

Methods

DeMo tackles outlier rejection by focusing on motion consensus, a prior that assumes the overall spatial transformation between two images should be smooth and consistent. The core of DeMo lies in its Deep Motion Field Consensus (DMFC) block, which interpolates the motion field in a high-dimensional space using learnable kernels within a Reproducing Kernel Hilbert Space (RKHS) framework. This allows for a flexible and efficient capture of global motion patterns.

Key novelties include:

  • RKHS-based Interpolation with Learnable Kernels: DeMo formulates motion field interpolation as an optimization problem in RKHS. Learnable deep kernels are used to capture complex relationships within the motion field, enabling robust global consensus.
  • Scene-Adaptive Sampling: To overcome the cubic computational complexity of explicit interpolation (O(N³)), DeMo introduces a scene-adaptive sampling strategy. This method implicitly selects more scene-representative motions, reducing complexity to nearly linear (O(N)) while preserving accuracy.
  • Local Consensus Complementation (LCC) Block: To handle discontinuities in the motion field caused by depth variations, an LCC block is designed. This block complements consensus between spatial and feature channels, helping to distinguish depth changes and prevent over-smoothing.

The overall pipeline involves an initial module to embed motion vectors into a high-dimensional space, followed by the LCC block for local refinement, then scene-adaptive sampling, and finally the DMFC block for global consensus interpolation. The rectified motion field is then used to predict inlier/outlier scores.

Datasets

DeMo was evaluated on several standard benchmarks:

  • YFCC100M: A large dataset for outdoor scenes, used for relative pose estimation and generalization ability tests.
  • SUN3D: A dataset for indoor scenes, also used for relative pose estimation.
  • HPatches: A benchmark for homography estimation, featuring scenes with illumination and viewpoint changes.
  • Aachen Day-Night: A benchmark for visual localization, specifically designed to test robustness to illumination changes.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Global Motion Consensus is Underrated: The paper strongly argues that global motion consensus is a powerful, yet under-explored, cue for robust correspondence learning. By explicitly modeling this global consistency, DeMo achieves superior performance.
  • Learnable Kernels for Flexibility: Using learnable kernels in RKHS allows the model to adaptively learn the relationships within the motion field, making it more flexible than fixed kernel approaches.
  • Scene-Adaptive Sampling is Crucial for Efficiency: The O(N³) complexity of traditional interpolation methods is a major bottleneck. The scene-adaptive sampling strategy is a clever way to achieve near-linear complexity without sacrificing accuracy, making the method more practical.
  • Local and Global Consensus Complement Each Other: The LCC block effectively addresses the limitations of global methods in handling depth discontinuities, showing that a combination of local and global reasoning is key.

Key Results

  • Superior Relative Pose Estimation: DeMo significantly outperforms state-of-the-art methods on both outdoor (YFCC100M) and indoor (SUN3D) datasets for relative pose estimation, achieving higher AUC scores across various error thresholds. For example, on YFCC100M at 5°, DeMo achieved 32.57/34.63 (weighted eight-point/RANSAC) compared to DeMatch's 30.89/32.98.
  • Strong Homography Estimation: DeMo demonstrates excellent performance on the HPatches benchmark, achieving higher accuracy and a better F-score for outlier rejection compared to other methods, while maintaining competitive computation times.
  • Robust Visual Localization: When integrated into the HLoc pipeline, DeMo shows improved performance on the Aachen Day-Night benchmark, particularly excelling in challenging night-time scenes.
  • Generalization: DeMo exhibits strong generalization ability across different descriptors (SIFT, RootSIFT, LIFT, SuperPoint) and datasets, indicating the robustness of its learned motion consensus prior.
  • Efficiency: The scene-adaptive sampling strategy reduces computational complexity significantly, making DeMo practical for real-time applications.

Limitations and Future Work

  • Computational Complexity: While significantly improved by scene-adaptive sampling, the method still has a computational cost. The paper mentions that increasing the sampling parameter M improves accuracy but also increases computation.
  • Parameter Tuning: The paper notes that the regularization parameter λ and the kernel parameter β are trainable, but the scene-adaptive sampling parameter M requires careful tuning for optimal performance-efficiency trade-off.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success in handling challenging scenarios like large depth differences and illumination variations suggests potential applications in more complex robotic perception tasks, autonomous driving, and dense 3D reconstruction. The learned kernels and consensus mechanisms could also be explored for other geometric reasoning tasks.

Summary updated

BibTeX
@inproceedings{lu2025demo,
  author = {Yifan Lu and Jiajun Le and Zizhuo Li and Yixuan Yuan and Jiayi Ma},
  title = {{DeMo: Deep Motion Field Consensus with Learnable Kernels for Two-view Correspondence Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32622},
}
Figure