CoMoDA: Continuous Monocular Depth Adaptation Using Past Experiences

Yevhen Kuznietsov, Marc Proesmans, Luc Van GoolWACV 2021 · DOI

Introduction

Monocular depth estimation—the task of predicting 3D scene geometry from a single 2D image—is a fundamental challenge in computer vision. While self-supervised learning has enabled models to learn depth from video sequences without manual labels, these models typically remain static after training. Consequently, they often struggle when deployed in novel environments with different lighting, weather, or camera configurations.

The Challenge of Test-Time Adaptation

Existing "test-time refinement" techniques attempt to adapt models on the fly. However, they face two primary hurdles:

  1. Computational Overhead: Many methods require numerous iterations per frame, making them too slow for real-time robotics or autonomous driving.
  2. Catastrophic Forgetting: Without proper regularization, models can overfit to the current visual scene, losing the general depth cues learned during initial training.

Methodology: Continuous Adaptation with CoMoDA

CoMoDA (Continuous Monocular Depth Adaptation) addresses these issues by shifting from isolated refinement to a continuous learning paradigm.

The CoMoDA framework architecture, highlighting the continuous optimization loop and the use of a replay buffer to stabilize learning.

Key Components

  • Continuous Learning Loop: Instead of resetting the model for each frame, CoMoDA performs a single gradient update per frame. This allows the model to accumulate knowledge across a sequence.
  • Replay Buffer: To prevent overfitting and maintain stability, each update step uses a mini-batch containing the current frame and random samples from the original training set (the "distant past").
  • Scale-Aware Loss: By integrating vehicle velocity data into the loss function, CoMoDA enables the prediction of absolute metric depth rather than just relative proportions.
  • Semantic Masking: To handle dynamic objects like cars and pedestrians that violate static-world assumptions, the framework uses semantic masks to ignore these regions during the update step.

Experimental Evaluation

The authors evaluated CoMoDA on two major benchmarks: KITTI (for intra-dataset adaptation) and NuScenes (for cross-dataset adaptation). The latter tests the model's ability to bridge significant domain shifts in real-time.

Results and Insights

CoMoDA achieves state-of-the-art accuracy on the KITTI Eigen split, outperforming more complex architectures. Notably, it runs at approximately 107ms per frame, which is 15x to 300x faster than previous test-time refinement methods.

Qualitative comparison of depth estimation results. CoMoDA produces sharper depth maps that are closer to the ground truth (GT) compared to non-adapted and other state-of-the-art methods.

The inclusion of a replay buffer proved to be the "secret sauce" for stability. Without it, adapting solely on the current video stream leads to erratic performance. Furthermore, the cross-dataset experiments on NuScenes demonstrated CoMoDA's ability to effectively learn the visual characteristics of a new city on the fly.

Conclusion and Future Work

CoMoDA demonstrates that deployment can be an extension of the learning process. By treating inference as a continuous adaptation task, AI models can "evolve" alongside the environments they navigate. Future research directions include improving the handling of complex dynamic objects and enforcing temporal smoothness through recurrent architectures.

Summary updated

BibTeX
@inproceedings{kuznietsov2021comoda,
  author = {Yevhen Kuznietsov and Marc Proesmans and Luc Van Gool},
  title = {{CoMoDA: Continuous Monocular Depth Adaptation Using Past Experiences}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00295},
}
Figure