MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos

Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, Noah Snavely

Introduction

Extracting 3D structure and camera motion from a video—a field known as Structure from Motion (SfM) or SLAM—is a cornerstone of computer vision. However, traditional methods are notoriously sensitive to scene dynamics. They usually require a static environment and significant camera parallax to function correctly. When applied to "casual" handheld videos—such as a busy street or a simple camera rotation—these systems often fail due to independent object motion or lack of depth cues.

Researchers from Google DeepMind, UC Berkeley, and the University of Michigan have introduced MegaSaM, a robust pipeline designed specifically for these challenging monocular videos. MegaSaM achieves state-of-the-art accuracy in estimating camera poses and depth maps by combining a differentiable bundle adjustment (BA) framework with learned motion masking and monocular depth priors.

MegaSaM reconstructs 3D structure and camera motion from casual videos with moving objects.

Methodology

MegaSaM builds upon DROID-SLAM, a deep visual SLAM framework, and introduces several key innovations to handle dynamic, casual footage:

  1. Learned Motion Probability Maps: The system includes a dedicated network that predicts which pixels belong to moving objects. These regions are automatically downweighted during the Bundle Adjustment (BA) process so they do not "pollute" the camera tracking.
  2. Two-Stage Training: To prevent unstable training behaviors, the authors use a two-step approach: first training on static scenes to master ego-motion, then freezing those weights to train the motion-masking module on dynamic scenes.
  3. Data-Driven Initialization: Instead of starting from scratch, MegaSaM uses off-the-shelf monocular depth models (like DepthAnything) to provide initial estimates for geometry and focal length.
  4. Uncertainty-Aware Global BA: The system utilizes the Hessian matrix from the optimization layer to quantify "epistemic uncertainty." If the camera is just rotating (low parallax), the system detects high uncertainty and leans more heavily on monocular depth priors to maintain stability.
  5. Consistent Video Depth Optimization: After tracking, the system performs a fast, first-order optimization to produce high-resolution, temporally smooth depth maps without needing expensive per-video neural network fine-tuning.
Ablation study demonstrating the impact of monocular depth initialization and uncertainty-aware bundle adjustment on reconstruction quality.

Key Results and Insights

MegaSaM was trained exclusively on synthetic data (TartanAir and Kubric) and evaluated on a variety of benchmarks, including MPI Sintel and real-world handheld videos (DyCheck). The system demonstrates significant improvements over prior and concurrent baselines:

  • Accuracy: On the Sintel dataset, MegaSaM reduced the Absolute Trajectory Error (ATE) by approximately 50% compared to previous state-of-the-art methods.
  • Robustness: It remains stable on "uncalibrated" videos where the focal length is unknown—a scenario where many traditional systems fail.
  • Speed: The system is highly efficient, providing high-quality results in a fraction of the time required by optimization-heavy methods (typically ~0.7s to 1.0s for camera tracking).

A key insight of the work is the use of the Hessian matrix as a guide. By mathematically quantifying uncertainty, the system intelligently decides when to rely on geometric constraints versus when to trust neural depth priors.

Limitations and Future Work

Despite its robustness, MegaSaM faces challenges when moving objects dominate the frame, as it may mistakenly treat the object as the static background. It also struggles with "selfie-style" co-linear motion and does not currently handle varying focal lengths (zooming) or extreme radial distortion. The authors suggest that incorporating even stronger vision foundation models into the pipeline is a promising path for future improvements.

Summary updated

Abstract

We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Most conventional structure from motion and monocular SLAM techniques assume input videos that feature predominantly static scenes with large amounts of parallax. Such methods tend to produce erroneous estimates in the absence of these conditions. Recent neural network-based approaches attempt to overcome these challenges; however, such methods are either computationally expensive or brittle when run on dynamic videos with uncontrolled camera motion or unknown field of view. We demonstrate the surprising effectiveness of a deep visual SLAM framework: with careful modifications to its training and inference schemes, this system can scale to real-world videos of complex dynamic scenes with unconstrained camera paths, including videos with little camera parallax. Extensive experiments on both synthetic and real videos demonstrate that our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work, with faster or comparable running times. See interactive results on our project page: https://mega-sam.github.io/
BibTeX
@inproceedings{li2025megasam,
  author = {Zhengqi Li and Richard Tucker and Forrester Cole and Qianqian Wang and Linyi Jin and Vickie Ye and Angjoo Kanazawa and Aleksander Holynski and Noah Snavely},
  title = {{MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=0paUxuagW6},
}
Figure