Abstract
Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.
Introduction
Monocular Depth Estimation (MDE)—predicting 3D depth from a single 2D image—has seen massive leaps recently with models like Depth Anything V2. However, these models are primarily designed for static images. When applied to video frame-by-frame, they suffer from "flickering" and "drifting," where the depth of an object changes unnaturally between adjacent frames.
Existing solutions to this temporal inconsistency typically fall into two categories:
- Geometric Priors: Utilizing optical flow or camera poses to align frames. These methods are often computationally expensive and sensitive to flow estimation errors.
- Generative Models: Leveraging video diffusion models. While these produce high-quality details, they are computationally intensive, slow, and generally limited to short clips (under 10 seconds).
Video Depth Anything (VDA) addresses these challenges by providing an accurate, temporally stable, and computationally efficient solution capable of processing super-long videos lasting several minutes.
Methodology
The architecture of Video Depth Anything is built upon the Depth Anything V2 foundation, incorporating three critical modifications to enable temporal consistency:
Spatio-Temporal Head (STH)
The standard DPT head is replaced with a new Spatio-Temporal Head containing temporal attention layers. To maintain the rich spatial features learned from millions of images, the backbone encoder is kept frozen during training. This allows the new head to focus exclusively on learning temporal relationships between features.
Temporal Gradient Matching (TGM) Loss
To ensure consistency without the overhead of optical flow, the authors introduced a Temporal Gradient Matching loss. This loss constrains the temporal depth gradient of the predictions to match that of the ground truth, effectively penalizing flickering by ensuring the "rhythm" of depth changes is preserved.
Long-Video Inference Strategy
To handle videos of arbitrary length, a segment-wise processing strategy was developed. Each segment is processed in conjunction with "key frames" from previous segments, providing a global anchor that prevents scale and shift drift over time.
Results and Evaluation
Video Depth Anything was trained on a massive joint dataset comprising 730,000 labeled video frames and 0.62 million unlabeled images for self-training. Evaluation across five major benchmarks (KITTI, ScanNet, Bonn, NYUv2, and Sintel) demonstrates its superior performance.
Performance and Efficiency
VDA achieved state-of-the-art results on 4 out of 5 benchmarks for spatial accuracy and significantly outperformed competitors in temporal consistency. Notably, the small version (VDA-S) achieves real-time performance at over 30 FPS (9.1ms latency) on an Nvidia A100 GPU. Even the large version remains highly efficient, with only a 10% latency increase over the original image-only model.
Super-Long Video Consistency
The model's robustness is highlighted by its performance on a 196-second continuous take of pair skating. Unlike diffusion-based or optimization-based methods, VDA maintains consistent depth estimation throughout the nearly 4,700 frames without quality degradation or drifting.
Conclusion
Video Depth Anything successfully extends the capabilities of large-scale monocular depth models to the video domain. By combining a lightweight spatio-temporal head with a novel temporal gradient loss and a key-frame inference strategy, it achieves unprecedented consistency for super-long videos. While there is still room for improvement in specific "cinematic" scenarios found in datasets like Sintel, VDA sets a new benchmark for efficient, high-quality video depth estimation.
Summary updated