Abstract
In this paper, we tackle the problem of learning Structure-from-Motion (SfM) through the use of graph attention networks. SfM is a classic computer vision problem typically solved through iterative minimization of reprojection errors, referred to as Bundle Adjustment (BA), starting from a good initialization. To obtain a sufficient initialization, conventional methods rely on a sequence of sub-problems (such as pairwise pose estimation, pose averaging, or triangulation). In this work, we replace these sub-problems by learning a model that takes 2D keypoints detected across multiple views as input and outputs the corresponding camera poses and 3D keypoint coordinates. Our model, GASfM, leverages graph neural networks to learn SfM-specific primitives, enabling fast inference for new and unseen sequences. Experimental results show that the proposed model outperforms competing learning-based methods and challenges COLMAP while maintaining a significantly lower runtime. Our code is available at https://github.com/lucasbrynte/gasfm/.
Introduction
Structure-from-Motion (SfM) is the process of estimating 3D structures and camera poses from a set of 2D images. The industry standard for this task is Bundle Adjustment (BA), a non-convex optimization process. However, BA is highly sensitive to initialization; poor starting estimates often lead to local minima and failed reconstructions. Traditional pipelines like COLMAP generate initializations through a complex sequence of geometric sub-problems, which are computationally expensive and slow.
While deep learning has shown promise in various vision tasks, previous "Deep SfM" attempts often struggled with generalization, requiring per-scene fine-tuning. This paper introduces GASfM (Graph Attention SfM), a learned model designed to provide high-quality, robust initializations for SfM in a fraction of the time required by traditional methods, without the need for scene-specific training.
Methodology
Graph Representation
GASfM treats SfM as a graph-based inference task. The scene is represented as a heterogeneous graph consisting of four node types: Views ($\mathcal{V}$), Scene Points ($\mathcal{S}$), Projections ($\mathcal{P}$), and a Global node ($g$). Projections serve as the edges connecting views to the 3D points they observe.
Iterative Geometric Reasoning
The model utilizes $L=12$ layers of Graph Attention (GATv2) to iteratively refine node features. These layers allow the network to "attend" to specific geometric relationships, effectively learning the underlying rules of multiview geometry. For example, a camera node can aggregate information from all visible 3D points to update its pose estimate.
Robustness through Training
To handle real-world noise and outliers, the authors employed two key strategies: random camera rotation augmentation and artificial outlier injection. Surprisingly, training with outliers acted as a powerful regularizer, improving the model's ability to identify and ignore noise in clean data, similar to the function of RANSAC in traditional pipelines.
Experimental Results
The model was evaluated on diverse real-world datasets, including Olsson’s and VGG datasets, focusing on its ability to generalize to completely novel scenes.
- Efficiency: GASfM is 5–10x faster than COLMAP. Inference typically takes less than a second, followed by a brief BA step that concludes in seconds rather than minutes.
- Accuracy: After final refinement, GASfM achieved a reprojection error of 0.85 pixels, closely approaching COLMAP's 0.66 pixels. In 8 out of 10 test scenes, GASfM actually produced lower rotation errors than the traditional baseline.
- Generalization: Unlike previous learning-based methods like DPESFM, GASfM generalizes effectively to unseen environments without any fine-tuning, demonstrating its ability to learn fundamental geometric primitives.
Conclusion
GASfM bridges the gap between traditional geometric SfM and deep learning. By replacing fragile initialization heuristics with a robust graph-based learned model, it achieves high-quality reconstructions with significant speed improvements. Future work includes integrating the model with learned feature matchers for a fully end-to-end differentiable SfM pipeline and scaling to massive datasets with thousands of images.
Summary updated