Generalized Differentiable RANSAC

Tong Wei, Yash Patel, Alexander Shekhovtsov, Jirí Matas, Daniel Barath

Introduction

Robust estimation is a cornerstone of computer vision, essential for tasks like camera pose estimation and 3D reconstruction where data is frequently contaminated by outliers. For decades, Random Sample Consensus (RANSAC) has served as the industry standard. However, its discrete nature—involving random sampling and non-differentiable inlier counting—has historically made it a "black box" that is difficult to integrate into end-to-end deep learning pipelines.

$\nabla$-RANSAC (Nabla-RANSAC) introduces the first generalized, fully differentiable RANSAC pipeline. By replacing discrete operations with continuous relaxations, the authors enable neural networks to directly optimize for final geometric metrics, such as pose accuracy, rather than relying on sub-optimal proxy losses.

The $\nabla$-RANSAC Architecture

The core innovation of $\nabla$-RANSAC lies in making every stage of the robust estimation process differentiable. The pipeline consists of three main components integrated into a cohesive learning framework.

Overview of the $\nabla$-RANSAC pipeline, illustrating the end-to-end differentiable flow from feature matching to loss calculation.
  1. Gumbel-Softmax Sampler: To overcome the non-differentiability of random point selection, the method employs a Gumbel-Softmax relaxation. This allows the network to learn an importance distribution over matches, effectively training the model to prioritize points most likely to yield accurate geometric models.
  2. Differentiable Geometric Solvers: The authors implemented a suite of standard solvers (5-point, 7-point, and 8-point algorithms) within PyTorch. This ensures that the model estimation step can propagate gradients back to the input features.
  3. Trainable Quality Function: Instead of simple inlier counting, $\nabla$-RANSAC uses a trainable function that marginalizes over multiple generated models. During training, the system minimizes a loss based on ground-truth pose error and epipolar distance.

Experimental Evaluation

The method was evaluated across diverse benchmarks, including PhotoTourism (outdoor scenes), ScanNet (indoor environments), and 3DMatch/3DLoMatch (3D point cloud registration). These tests covered fundamental (F) and essential (E) matrix estimation, as well as 3D registration.

Key Insights

A significant finding of the study is the synergy between minimal solvers and training stability. The authors observed that using minimal solvers (like the 5-point algorithm) during training is superior to over-determined ones (like the 8-point algorithm), as they are less susceptible to degenerate configurations. Furthermore, $\nabla$-RANSAC can be used to fine-tune existing feature matchers like LoFTR, significantly boosting their performance on real-world datasets by aligning them with the final geometric task.

Results and Performance

$\nabla$-RANSAC achieves state-of-the-art accuracy across major benchmarks. On the PhotoTourism dataset, it consistently outperformed established methods like MAGSAC++ and NG-RANSAC.

Comparison of $\nabla$-RANSAC against state-of-the-art robust estimators on the PhotoTourism dataset for Essential matrix estimation.

As shown in the results, $\nabla$-RANSAC achieves a leading AUC@5° of 0.41 for Essential matrix estimation. Beyond accuracy, the method maintains competitive inference speeds in the tens of milliseconds, making it practical for real-time applications. It also demonstrated a ~5% improvement in F1 scores for fundamental matrix estimation, highlighting its superior robustness to outliers.

Conclusion and Future Directions

$\nabla$-RANSAC represents a significant leap in bridging the gap between classical geometry and deep learning. While the implementation of differentiable polynomial solvers requires careful numerical handling, the benefits of end-to-end optimization are clear. Future work may explore even more sophisticated differentiable ranking methods, such as NeuralSort, to further refine the sampling process and extend the framework to even more complex geometric tasks.

The code and pre-trained models are available at: https://github.com/weitong8591/differentiable_ransac

Summary updated

Abstract

We propose $\nabla$-RANSAC, a generalized differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. The proposed approach enables the use of relaxation techniques for estimating the gradients in the sampling distribution, which are then propagated through a differentiable solver. The trainable quality function marginalizes over the scores from all the models estimated within $\nabla$-RANSAC to guide the network learning accurate and useful inlier probabilities or to train feature detection and matching networks. Our method directly maximizes the probability of drawing a good hypothesis, allowing us to learn better sampling distributions. We test $\nabla$-RANSAC on various real-world scenarios on fundamental and essential matrix estimation, and 3D point cloud registration, outdoors and indoors, with handcrafted and learning-based features. It is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives. The code and trained models are available at https://github.com/weitong8591/differentiable_ransac.
BibTeX
@inproceedings{wei2023generalized,
  author = {Tong Wei and Yash Patel and Alexander Shekhovtsov and Jirí Matas and Daniel Barath},
  title = {{Generalized Differentiable RANSAC}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01618},
}
Figure