TL;DR
The authors introduce ORBIT, a challenging new benchmark for evaluating Structure-from-Motion (SfM) algorithms in complex, real-world scenarios. By leveraging 360° videos to establish robust pseudo-ground truth trajectories and reprojecting them into narrow perspective clips, the benchmark reveals that all current state-of-the-art SfM methods completely fail on at least 36% of these real-world sequences.
What Problem Does This Paper Address?
Structure-from-Motion (SfM)—the process of reconstructing 3D scene geometry and camera trajectories from 2D image sequences—is foundational for robotics, AR/VR, and generative 3D world models. However, current SfM algorithms frequently break down when faced with real-world complexities like dynamic objects, textureless surfaces, or rapid camera movements.
Compounding this issue is a circular benchmarking problem: evaluating these algorithms requires high-quality ground-truth camera poses, but obtaining them in the wild is incredibly difficult. Many existing datasets rely on traditional SfM tools like COLMAP to generate "ground truth," which fails to expose the limitations of those very tools. Synthetic datasets lack realism, and sensor-based datasets (using GPS/IMUs) are restricted to highly controlled environments.
Methods
To break this loop, the authors exploit a clever insight: while a standard narrow field-of-view (FOV) camera can easily lose track in a challenging scene, a 360° panoramic camera captures the entire environment. Even if a dynamic object or a blank wall occludes one direction, stable features in other directions remain visible to anchor the camera's pose.
The pipeline consists of:
- Rig-Based Pose Estimation: 360° equirectangular video frames are reprojected into four overlapping 120° FOV "cube faces" (front, back, left, right), treating them as a rigid multi-camera rig. A custom bundle adjustment pipeline initializes with ORB-SLAM2 and refines poses using SIFT feature matching across all views.
- Metric Scale Calibration: Because SfM inherently lacks absolute scale, the authors use Depth Pro, a zero-shot monocular metric depth model, to predict depth maps and scale the SfM trajectories to approximate metric scale.
- Cross-Validation: To ensure the pseudo-ground truth is flawless, they run ORB-SLAM2 independently on different perspective crops. If these independent runs disagree with the 360° rig trajectory beyond strict thresholds, the clip is discarded.
- Benchmark Generation: Finally, they generate perspective-view evaluation clips by cropping the 360° videos. They intentionally introduce synthetic camera motions (like panning, tilting, and handheld jitter) and prioritize starting views where standard perspective SLAM failed.
Datasets
- ORBIT Benchmark (New): Contains 308 perspective clips extracted from 77 unique 360° videos collected from the web. Clips range from 150 to 1,000 frames at 30 fps, featuring diverse challenges like crowds, water, snow, and high-speed motion.
- 360Loc & Synthetic 360: Used to validate the fidelity of the 360° ground-truth pipeline. 360Loc provides real-world LiDAR ground truth, while Synthetic 360 consists of photorealistic Blender renders.
Key Insights and Ideas
- Omnidirectional Context as a Ground-Truth Engine: 360° video allows researchers to reliably solve SfM globally, creating a high-fidelity benchmark for narrow-FOV perspective videos where traditional SfM fails completely.
- Architectural Failure Modes: Optimization-based methods (COLMAP, MegaSaM) and feed-forward neural networks (VGGT, MonST3R) fail in fundamentally different ways. Optimization methods are highly accurate when they work but suffer from catastrophic failures or drop frames entirely. Feed-forward methods always output a trajectory but rarely match the ground truth precisely.
- Dataset Bias in Feed-Forward Models: Feed-forward models like VGGT-Long performed significantly worse when synthetic camera rotations were introduced compared to the original video trajectories. This suggests that current training sets for these models lack sufficient camera rotation diversity.
Key Results
- Pipeline Fidelity: The 360° rig pipeline achieved an Absolute Trajectory Error (ATE) of just 0.07m on the real-world 360Loc dataset and 0.0m on the Synthetic 360 dataset, proving its reliability as a pseudo-ground truth generator.
- Benchmark Performance: Every single tested method failed on at least 36% of the ORBIT benchmark.
- COLMAP vs. Deep Learning: Traditional COLMAP failed on 56.49% of the clips, struggling heavily with crowded scenes and textureless regions. MegaSaM (combined with RoMo motion segmentation) performed the best but still failed on 38.56% of the clips. Feed-forward models like MonST3R and VGGT-Long exhibited failure rates exceeding 94% under strict evaluation thresholds.
Limitations and Future Work
- Fixed Intrinsics: The current benchmark assumes known, fixed camera intrinsics for each perspective crop. Real-world in-the-wild videos often feature unknown or changing focal lengths (zooming). The authors plan to introduce variable focal lengths in future iterations of ORBIT to challenge intrinsics prediction.
- Data Sourcing: The pipeline relies on the availability of high-quality 360° videos, which, while growing in popularity, are less ubiquitous than standard perspective videos.
Summary updated