Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency

Seokju Lee, Sunghoon Im, Stephen Lin, In So Kweon

Abstract

We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion, and depth in a monocular camera setup without supervision. Our technical contributions are three-fold. First, we highlight the fundamental difference between inverse and forward projection while modeling the individual motion of each rigid object, and propose a geometrically correct projection pipeline using a neural forward projection module. Second, we design a unified instance-aware photometric and geometric consistency loss that holistically imposes self-supervisory signals for every background and object region. Lastly, we introduce a general-purpose auto-annotation scheme using any off-the-shelf instance segmentation and optical flow models to produce video instance segmentation maps that will be utilized as input to our training pipeline. Through extensive experiments conducted on the KITTI and Cityscapes dataset, our framework is shown to outperform the state-of-the-art depth and motion estimation methods.

Introduction: The Challenge of Dynamic Scenes

Unsupervised monocular depth estimation typically relies on the "rigidity assumption," treating depth learning as a view synthesis problem where the scene is static and only the camera moves. However, in real-world urban environments, this assumption fails. When objects like cars or pedestrians move independently, traditional inverse warping techniques produce significant "ghosting" artifacts and distortions. While many models simply mask these regions as noise, this research treats moving objects as vital geometric signals to be modeled explicitly.

Comparison of warping techniques: traditional inverse warping often results in 'ghosting' artifacts (left) where moving objects are duplicated or smeared, a problem addressed by the proposed instance-aware framework.

Methodology

Neural Forward Projection

Unlike standard inverse warping, which looks back from a target frame to a source frame, the authors propose a Neural Forward Projection module. Forward warping is geometrically more accurate for moving objects but often leaves "holes" in the reconstructed image. The proposed differentiable module uses a neural upsampling technique to fill these gaps, ensuring a dense and clean reconstruction for both static and dynamic regions.

Instance-Aware Consistency

The framework decomposes the scene into a static background and individual object instances. By applying specific photometric and geometric consistency losses to each region, the model ensures that an object's predicted 3D shape and 6-DoF motion are consistent across frames. To facilitate this without manual labels, the authors developed an Auto-Annotation Scheme that leverages off-the-shelf instance segmentation and optical flow models to generate video instance masks automatically.

The framework utilizes an auto-annotation scheme to generate instance masks, allowing the model to apply specific consistency losses to individual dynamic objects.

Key Results and Insights

The proposed method, Insta-DM, demonstrates that modeling moving objects as signal rather than noise significantly improves overall depth accuracy. On the KITTI Eigen split, the model reduced the Absolute Relative (AbsRel) error in dynamic regions from approximately 0.396 in baseline models to 0.178.

  • Forward vs. Inverse: Forward projection preserves the appearance of moving objects far better than inverse warping, eliminating the "smearing" effect common in previous state-of-the-art models.
  • Scale Consistency: By implementing a height constraint loss, the model avoids the "infinite depth" trap, where objects moving at the same speed as the camera are incorrectly predicted to be at an infinite distance.
  • Visual Quality: Qualitative evaluations show sharper object boundaries and a complete absence of the ghosting artifacts that typically plague unsupervised view-synthesis methods.

Conclusion

By explicitly modeling the 6-DoF motion of individual objects through a geometrically correct forward projection pipeline, this framework sets a new standard for unsupervised monocular depth estimation in complex, dynamic environments. While currently limited by the quality of external segmenters and the assumption of rigid motion within instances, it provides a robust foundation for future research into non-rigid motion and integrated segmentation-depth networks.

Summary updated

Abstract

We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion, and depth in a monocular camera setup without supervision. Our technical contributions are three-fold. First, we highlight the fundamental difference between inverse and forward projection while modeling the individual motion of each rigid object, and propose a geometrically correct projection pipeline using a neural forward projection module. Second, we design a unified instance-aware photometric and geometric consistency loss that holistically imposes self-supervisory signals for every background and object region. Lastly, we introduce a general-purpose auto-annotation scheme using any off-the-shelf instance segmentation and optical flow models to produce video instance segmentation maps that will be utilized as input to our training pipeline. These proposed elements are validated in a detailed ablation study. Through extensive experiments conducted on the KITTI and Cityscapes dataset, our framework is shown to outperform the state-of-the-art depth and motion estimation methods. Our code, dataset, and models are publicly available.
BibTeX
@inproceedings{lee2021learning,
  author = {Seokju Lee and Sunghoon Im and Stephen Lin and In So Kweon},
  title = {{Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=uO9Ytb-cQAT},
}
Figure