Learning Accurate Dense Correspondences and When To Trust Them

Prune Truong, Martin Danelljan, Luc Van Gool, Radu Timofte

Abstract

Establishing dense correspondences between a pair of images is an important and general problem. However, dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the task of pose estimation.

Introduction

Finding "dense correspondences"—matching every pixel in one image to its counterpart in another—is a cornerstone of computer vision, essential for 3D reconstruction, autonomous driving, and image editing. However, current models often struggle with large displacements, lighting changes, or textureless regions. A critical failure mode is the inability of networks to signal when they are uncertain, often leading to "confident hallucinations" in occluded or featureless areas. This work introduces PDC-Net to address this by simultaneously estimating accurate flow and a reliable uncertainty map.

Methodology

PDC-Net treats correspondence as a probabilistic regression problem, predicting the parameters of a probability distribution rather than a single flow vector.

Constrained Mixture Model (CMM): Standard models using single distributions struggle to represent both highly accurate inliers and complete outliers. PDC-Net employs a mixture of distributions with constrained variance ranges. This approach breaks permutation invariance, leading to more stable learning and better separation of error types.

Figure 6: The distribution of absolute error, highlighting the separation between inliers (red) and outliers (blue) achieved by the mixture model.

Architecture and Training: The network utilizes a Correlation Uncertainty Module that processes 2D slices of the correlation volume independently, assessing match reliability based on local feature uniqueness. To ensure generalizability, the model is trained using Self-Supervised Perturbations. By adding local flow noise to synthetic warps, the network is forced to learn actual pixel appearance rather than relying on global smoothness priors.

Figure 4: A visualization of the predicted confidence map, where the network identifies reliable regions for matching.

Experimental Results

The model was evaluated across diverse benchmarks including MegaDepth, RobotCar, ETH3D, and KITTI.

  • State-of-the-Art Accuracy: On MegaDepth, PDC-Net achieved a PCK-1 of 71.81%, a significant improvement over the previous state-of-the-art (~53%).
  • Generalization: Without fine-tuning, PDC-Net outperformed specialized models like RAFT on the KITTI-2015 training set (AEPE of 5.22 vs 5.54).
  • Robust Pose Estimation: In YFCC100M two-view geometry tasks, PDC-Net was 12.2x faster than RANSAC-Flow while maintaining superior accuracy.

Discussion and Conclusion

A key insight of this work is that uncertainty acts as a powerful filter. By discarding low-confidence matches, the accuracy of downstream tasks like 3D reconstruction is drastically improved. While the model handles ambiguity well, extreme perspective shifts still benefit from iterative multi-stage processing. Future work involves tighter integration of these probabilistic maps into bundle adjustment phases of 3D pipelines. PDC-Net sets a new standard for robust dense correspondence by knowing exactly when to trust its own predictions.

Summary updated

Abstract

Establishing dense correspondences between a pair of images is an important and general problem. However, dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the task of pose estimation. Code and models are available at https://github.com/PruneTruong/PDCNet.
BibTeX
@inproceedings{truong2021learning,
  author = {Prune Truong and Martin Danelljan and Luc Van Gool and Radu Timofte},
  title = {{Learning Accurate Dense Correspondences and When To Trust Them}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=Ghv0sg5Ofk0},
}
Figure