Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation

Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, Luigi Di Stefano

Introduction

Estimating depth from a single image (monocular depth estimation) is a classic "ill-posed" problem in computer vision: a single 2D image could technically represent an infinite number of 3D scenes. While humans use context and experience to solve this, computers traditionally rely on two methods: supervised learning, which requires expensive LiDAR ground truth, and unsupervised learning, which uses geometry (like stereo image pairs) as a training signal. However, unsupervised models often struggle with fine details, thin structures, and reflective surfaces. This paper explores whether semantic information—knowing that a group of pixels is a "car" or a "tree"—can guide and improve unsupervised geometric depth estimation.

Methodology

The researchers propose a semi-supervised framework using a "Y-shaped" encoder-decoder architecture. A single backbone (VGG or ResNet50) extracts features from the input image, which are then processed by two separate decoders: one for depth maps and another for semantic segmentation maps.

The proposed Y-shaped architecture featuring a shared encoder and dual decoders for depth and semantics.

The core innovation is the Cross-Domain Discontinuity Loss ($L_{cdd}$). Standard depth models often produce blurry edges where an object meets the background. The $L_{cdd}$ loss explicitly forces depth edges to align with semantic boundaries. If the segmentation map identifies an object boundary at a specific pixel, the depth map is encouraged to have a sharp transition there as well. The training is semi-supervised: depth is learned unsupervised via stereo-pair warping, while semantics is supervised using standard pixel-level category labels.

Key Insights

  • Semantics as a Geometric Prior: Objects of the same class (e.g., cars) tend to have similar 3D shapes and occupy specific "depth layers" relative to the road. By recognizing the object, the network develops a better intuition for its distance.
  • Solving the "Window" Problem: Geometric-only models often fail on transparent or reflective surfaces like car windows because the warping logic breaks down. By recognizing these pixels belong to a "car," the semantic branch helps the depth branch "fill in" the correct distance.
  • Edge Alignment: The cross-domain loss uses high-contrast boundaries in semantic maps to "clip" the bleeding edges often seen in unsupervised depth maps, resulting in crisper reconstructions of thin objects like poles and signs.

Results and Discussion

The inclusion of semantics led to measurable improvements across all standard metrics on the KITTI dataset. The full model achieved an Absolute Relative (Abs Rel) error of 0.136, compared to the baseline unsupervised model's 0.148.

Qualitative results show that the proposed method preserves the structure of complex objects (like guardrails and poles) far better than purely geometric methods. Even with a simpler VGG backbone, the semantically-aware model outperformed a more complex ResNet50 model that lacked semantic guidance.

Conclusion

This research demonstrates that "knowing what you see" significantly improves "knowing how far it is." By training a single neural network to perform both semantic segmentation and monocular depth estimation, semantic category labels act as a powerful prior to sharpen object boundaries and correct errors in unsupervised depth maps. Future work may explore using monocular video sequences instead of stereo pairs to further increase flexibility for real-world deployment.

Summary updated

BibTeX
@inproceedings{ramirez2018geometry,
  author = {Pierluigi Zama Ramirez and Matteo Poggi and Fabio Tosi and Stefano Mattoccia and Luigi Di Stefano},
  title = {{Geometry Meets Semantics for Semi-supervised Monocular Depth Estimation}},
  booktitle = {ACCV},
  year = {2018},
  doi = {10.1007/978-3-030-20893-6\_19},
}
Figure