Learning Structure-From-Motion with Graph Attention Networks

Lucas Brynte, José Pedro Iglesias, Carl Olsson, Fredrik KahlCVPR 2023 · OpenReview

Abstract

In this paper, we tackle the problem of learning Structure-from-Motion (SfM) through the use of graph attention networks. SfM is a classic computer vision problem typically solved through iterative minimization of reprojection errors, referred to as Bundle Adjustment (BA), starting from a good initialization. To obtain a sufficient initialization, conventional methods rely on a sequence of sub-problems (such as pairwise pose estimation, pose averaging, or triangulation). In this work, we replace these sub-problems by learning a model that takes 2D keypoints detected across multiple views as input and outputs the corresponding camera poses and 3D keypoint coordinates. Our model, GASfM, leverages graph neural networks to learn SfM-specific primitives, enabling fast inference for new and unseen sequences. Experimental results show that the proposed model outperforms competing learning-based methods and challenges COLMAP while maintaining a significantly lower runtime. Our code is available at https://github.com/lucasbrynte/gasfm/.

Introduction

Structure-from-Motion (SfM) is the process of estimating 3D structures and camera poses from a set of 2D images. The industry standard for this task is Bundle Adjustment (BA), a non-convex optimization process. However, BA is highly sensitive to initialization; poor starting estimates often lead to local minima and failed reconstructions. Traditional pipelines like COLMAP generate initializations through a complex sequence of geometric sub-problems, which are computationally expensive and slow.

While deep learning has shown promise in various vision tasks, previous "Deep SfM" attempts often struggled with generalization, requiring per-scene fine-tuning. This paper introduces GASfM (Graph Attention SfM), a learned model designed to provide high-quality, robust initializations for SfM in a fraction of the time required by traditional methods, without the need for scene-specific training.

Methodology

Graph Representation

GASfM treats SfM as a graph-based inference task. The scene is represented as a heterogeneous graph consisting of four node types: Views ($\mathcal{V}$), Scene Points ($\mathcal{S}$), Projections ($\mathcal{P}$), and a Global node ($g$). Projections serve as the edges connecting views to the 3D points they observe.

The heterogeneous graph structure of GASfM, illustrating the relationships between Views, Projections, Scene Points, and the Global node.

Iterative Geometric Reasoning

The model utilizes $L=12$ layers of Graph Attention (GATv2) to iteratively refine node features. These layers allow the network to "attend" to specific geometric relationships, effectively learning the underlying rules of multiview geometry. For example, a camera node can aggregate information from all visible 3D points to update its pose estimate.

Formal procedure for updating scene point features ($\mathcal{S}$) using Graph Cross-Attention from projection features ($\mathcal{P}$).

Robustness through Training

To handle real-world noise and outliers, the authors employed two key strategies: random camera rotation augmentation and artificial outlier injection. Surprisingly, training with outliers acted as a powerful regularizer, improving the model's ability to identify and ignore noise in clean data, similar to the function of RANSAC in traditional pipelines.

Experimental Results

The model was evaluated on diverse real-world datasets, including Olsson’s and VGG datasets, focusing on its ability to generalize to completely novel scenes.

  • Efficiency: GASfM is 5–10x faster than COLMAP. Inference typically takes less than a second, followed by a brief BA step that concludes in seconds rather than minutes.
  • Accuracy: After final refinement, GASfM achieved a reprojection error of 0.85 pixels, closely approaching COLMAP's 0.66 pixels. In 8 out of 10 test scenes, GASfM actually produced lower rotation errors than the traditional baseline.
  • Generalization: Unlike previous learning-based methods like DPESFM, GASfM generalizes effectively to unseen environments without any fine-tuning, demonstrating its ability to learn fundamental geometric primitives.

Conclusion

GASfM bridges the gap between traditional geometric SfM and deep learning. By replacing fragile initialization heuristics with a robust graph-based learned model, it achieves high-quality reconstructions with significant speed improvements. Future work includes integrating the model with learned feature matchers for a fully end-to-end differentiable SfM pipeline and scaling to massive datasets with thousands of images.

Summary updated

Abstract

In this paper we tackle the problem of learning Structure-from-Motion (SfM) through the use of graph attention networks. SfM is a classic computer vision problem that is solved though iterative minimization of reprojection errors, referred to as Bundle Adjustment (BA), starting from a good initialization. In order to obtain a good enough initial-ization to BA, conventional methods rely on a sequence of sub-problems (such as pairwise pose estimation, pose averaging or triangulation) which provide an initial solution that can then be refined using BA. In this work we re-place these sub-problems by learning a model that takes as input the 2D keypoints detected across multiple views, and outputs the corresponding camera poses and 3D key-point coordinates. Our model takes advantage of graph neural networks to learn SfM-specijic primitives, and we show that it can be used for fast inference of the reconstruction for new and unseen sequences. The experimental results show that the proposed model outperforms com-peting learning-based methods, and challenges COLMAP while having lower runtime. Our code is available at: https://github.com/lucasbrynte/gasfm/.
BibTeX
@inproceedings{brynte2023learning,
  author = {Lucas Brynte and José Pedro Iglesias and Carl Olsson and Fredrik Kahl},
  title = {{Learning Structure-From-Motion with Graph Attention Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=bIPz34WQKm},
}
Figure