Uncalibrated Structure from Motion on a Sphere

Jonathan Ventura, Viktor Larsson, Fredrik KahlICCV 2025

TL;DR

This paper tackles the challenge of reconstructing 3D scenes from images captured with a camera moving on a sphere, a common scenario for smartphone panoramas and specialized rigs. They introduce a novel global Structure-from-Motion (SFM) pipeline that accurately handles uncalibrated cameras (unknown focal length) and can even tolerate deviations from perfect spherical motion, outperforming existing methods.

What Problem Does This Paper Address?

Structure-from-Motion (SFM) is a fundamental computer vision task that aims to reconstruct the 3D structure of a scene and the camera poses that captured it. Traditional SFM pipelines often struggle with specific camera motion patterns, particularly when the camera moves along the surface of an imaginary sphere with its optical axis pointing outwards. This "spherical motion" is common in everyday scenarios like taking panoramic photos with a smartphone or using multi-camera rigs. Existing methods like COLMAP and GLOMAP frequently fail to produce accurate reconstructions in these cases, often misinterpreting the motion or yielding incomplete results. This paper addresses this gap by developing an SFM method specifically designed for uncalibrated spherical motion.

Methods

The core of the paper's contribution lies in its novel global SFM pipeline for uncalibrated spherical motion. Key aspects include:

  • Theoretical Analysis: The authors provide a theoretical analysis of uncalibrated spherical two-view and multi-view geometry. They demonstrate that while two-view spherical motion is a critical case for self-calibration (meaning it's inherently ambiguous), self-calibration is always possible with three or more distinct viewpoints.
  • Uncalibrated Spherical SFM Pipeline:
    1. Pairwise Fundamental Matrix Estimation: They leverage a robust spherical essential matrix solver (originally for calibrated cameras) to estimate fundamental matrices between image pairs, using LO-RANSAC for outlier rejection.
  • Camera Initialization and Refinement: A crucial step involves initializing the focal length and camera rotations. They analyze the relationship between focal length and relative rotation for spherical motion and devise a cost function to find the focal length that maximizes agreement among pairwise rotations. This is followed by iterative non-linear optimization to refine both the focal length and rotations.
  • Point Triangulation and Bundle Adjustment: After initializing poses and focal length, 3D points are triangulated. They then apply a two-stage bundle adjustment process: first, with a spherical motion constraint (fixing translation along the optical axis), and then a second time without this constraint to allow for general 3D reconstruction, even when the motion deviates from perfect sphericality.

Datasets

The authors introduce a new dataset called PhoneSweep, consisting of thirteen sequences captured by smartphones (iPhone 13 and Nexus 5X) with outward-facing spherical motion. They also evaluate their method on the Deep View Video dataset, which uses a hemi-spherical rig and provides metric ground truth for camera extrinsics and intrinsics. For both datasets, they generate pseudo-ground truth by using traditional SFM methods on general motion sequences and then localizing the spherical motion sequences within those reconstructions.

Key Insights and Ideas

  • Spherical Fundamental Matrices are Interchangeable with Essential Matrices: A key theoretical insight is that spherical fundamental matrices are equivalent to spherical essential matrices under a specific focal length. This allows them to adapt existing calibrated solvers for uncalibrated scenarios.
  • Self-Calibration is Possible Beyond Two Views: Unlike the critical case of two-view spherical motion, the paper proves that self-calibration of the focal length is always achievable with three or more distinct viewpoints.
  • Traditional SFM Fails on Spherical Motion: The paper highlights that standard SFM methods struggle with spherical motion, often misinterpreting the motion direction (inward vs. outward) or producing incomplete reconstructions.
  • Robustness to Deviations: The proposed method's ability to remove the spherical constraint in the final bundle adjustment step allows it to handle real-world scenarios where the motion isn't perfectly spherical.

Key Results

  • Superior Accuracy: The proposed method consistently outperforms COLMAP, GLOMAP, and RadialSFM in terms of accuracy metrics (Relative Rotation Accuracy, Relative Translation Accuracy, AUC) on both the PhoneSweep and Deep View Video datasets.
  • Reliable Reconstruction: Qualitative results (Figure 4) show that their method produces complete and accurate reconstructions where other methods fail, often due to misinterpreting the motion direction.
  • Competitive Speed: On the PhoneSweep dataset, their method is the second fastest, and on the Deep View Video dataset, it is the fastest.
  • Improved Focal Length Estimation: The paper demonstrates the effectiveness of their focal length estimation strategy, which is crucial for uncalibrated reconstruction.

Limitations and Future Work

  • Limitations: The authors note that in some cases on the Deep View Video dataset, while rotations were accurately estimated, camera positions had errors, suggesting that insufficient feature matches might limit translation accuracy. They also mention that their method, when restricted to a perfect spherical configuration (without the final general BA step), shows improved translation accuracy.
  • Future Work: The authors suggest future research directions including exploring alternative optimization strategies, extending the method to handle radial distortion, and investigating learning-based approaches as a complementary direction.

Summary updated

BibTeX
@inproceedings{ventura2025uncalibrated,
  author = {Jonathan Ventura and Viktor Larsson and Fredrik Kahl},
  title = {{Uncalibrated Structure from Motion on a Sphere}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure