Abstract
Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.
Introduction
Monocular depth estimation (MDE) has seen significant progress with the emergence of foundation models like Depth Anything V2. While these models excel at predicting depth from static images, they often struggle with temporal consistency when applied to video sequences. This results in "flickering" or sudden jumps in depth values for the same object across consecutive frames, which is detrimental for downstream applications like AR/VR or autonomous driving.
Existing approaches to solve this flickering problem typically rely on slow optimization-based tuning or computationally expensive diffusion-based models, which are often restricted to short clips of less than 10 seconds. Video Depth Anything (VDA) overcomes these limitations, providing a robust and efficient framework for consistent depth estimation in videos lasting several minutes.
Methodology
Video Depth Anything builds upon the Depth Anything V2 architecture, leveraging its powerful pre-trained ViT encoder. To adapt this image foundation model for video, the authors introduced several key architectural and training innovations.
Spatio-Temporal Head (STH) and TGM Loss
The standard DPT head is replaced with a lightweight Spatio-Temporal Head (STH) containing four temporal attention layers. These layers enable the model to capture temporal dependencies across frames. To train this head without the need for complex and error-prone optical flow or camera poses, the authors proposed a Temporal Gradient Matching (TGM) Loss. This loss constrains the temporal depth gradient of predictions to match the ground truth, ensuring smooth transitions between frames.
Key-Frame Based Inference
A major challenge in long video depth estimation is "scale drift," where the global depth scale shifts over time. VDA addresses this through a novel segment-wise inference strategy. By processing overlapping segments and referencing "key-frames" from previous segments, the model anchors the depth scale, ensuring global consistency across thousands of frames.
Experimental Results
The model was trained on a hybrid dataset of 0.55 million synthetic video frames and 0.62 million unlabeled real images. Evaluation across five major benchmarks (KITTI, ScanNet, Bonn, NYUv2, and Sintel) demonstrates that VDA sets a new state-of-the-art in zero-shot video depth estimation.
Performance and Efficiency
VDA offers a superior trade-off between accuracy, temporal consistency, and inference speed. The large model (VDA-L) achieves high accuracy with a latency of only 67ms per frame. The small version (VDA-S) is capable of real-time performance at 30 FPS on an Nvidia A100, significantly outperforming diffusion-based competitors like DepthCrafter, which requires over 900ms per frame.
Conclusion
Video Depth Anything successfully transforms a static image depth foundation model into a high-performance video depth estimator. By combining a lightweight temporal head with a gradient-based consistency loss and a robust long-video inference strategy, VDA achieves state-of-the-art results with high efficiency. Future work will explore unfreezing the backbone for joint spatio-temporal fine-tuning and optimizing the inference pipeline for true online streaming applications.
Summary updated