Contrastive Learning for Depth Prediction

Rizhao Fan, Matteo Poggi, Stefano MattocciaWorkshops @ CVPR 2023 · OpenReview

Abstract: Depth prediction is at the core of several computer vision applications, such as autonomous driving and robotics. It is often formulated as a regression task in which depth values are estimated through network layers. Unfortunately, the distribution of values on depth maps is seldom explored. Therefore, this paper proposes a novel framework combining contrastive learning and depth prediction, allowing us to pay more attention to depth distribution and consequently enabling improvements to the overall estimation process. Purposely, we propose a window-based contrastive learning module, which partitions the feature maps into non-overlapping windows and constructs contrastive loss within each one. Forming and sorting positive and negative pairs, then enlarging the gap between the two in the representation space, constraints depth distribution to fit the feature of the depth map. Experiments on KITTI and NYU datasets demonstrate the effectiveness of our framework.

Introduction

Depth prediction is a fundamental challenge in computer vision, essential for tasks ranging from autonomous navigation to augmented reality. Traditionally, this is treated as a per-pixel regression problem where a neural network predicts a continuous distance value for every coordinate in an image. However, standard regression losses often fail to capture the structural and statistical properties of the scene.

A key observation made by the authors is that "weak" depth models often produce unnatural depth distributions. For instance, instead of smooth gradients on a road or sharp jumps at object boundaries, a poor model might produce "peaky" histograms where values are clustered into narrow, unrealistic ranges. This leads to visual artifacts like blurry edges and "staircase" effects on flat surfaces.

Theoretical depth distributions for different surface geometries: (a) slanted plane, (b) fronto-parallel plane, (c) depth discontinuity.

Methodology: Window-based Contrastive Learning (WCL)

To address these distributional issues, the authors introduce the Window-based Contrastive Learning (WCL) module. Contrastive learning typically aims to pull similar representations together and push dissimilar ones apart. While powerful, applying this globally across high-resolution depth maps is computationally prohibitive.

The WCL module solves this through a localized approach:

  1. Window Partitioning: The feature map is divided into non-overlapping $N \times N$ windows (e.g., $7 \times 7$).
  2. Feature Pairing: Within each window, a similarity map is calculated. The module identifies the top 20% most similar features as "positive pairs" and the bottom 20% as "negative pairs."
  3. Contrastive Loss ($L_{closs}$): This loss function is applied to maximize the distance between positive and negative pairs in the representation space, effectively "stretching" the local feature distribution to better reflect the underlying geometry.
  4. Integration: WCL is a plug-and-play module inserted into the decoder of existing depth-prediction networks, acting as a training-time regularizer.

Experimental Results

The framework was evaluated on the KITTI (outdoor driving) and NYUv2 (indoor) datasets. The addition of WCL consistently improved performance across different baseline architectures.

  • Depth Completion (KITTI): Integrating WCL into the MSG-CHN baseline reduced the Root Mean Squared Error (RMSE) from 820.145mm to 810.273mm.
  • Monocular Depth Estimation (NYUv2): The BTS-ResNet50 model saw its RMSE improve from 0.419 to 0.409.

Qualitatively, the WCL module results in significantly sharper object boundaries and more realistic depth transitions. As shown in the comparison below, models equipped with WCL produce depth histograms that more closely match the ground truth distribution, avoiding the "peaky" artifacts of baseline models.

Qualitative comparison of depth maps and their corresponding value distributions across different models.

Conclusion

By shifting the focus from simple per-pixel regression to the statistical distribution of depth values, the WCL module provides a robust way to improve depth prediction accuracy and visual quality. While the method introduces some training-time overhead and sensitivity to window size, its ability to be discarded during inference makes it a highly practical enhancement for real-time applications. Future work may explore applying similar window-based contrastive techniques to other dense prediction tasks like semantic segmentation.

Summary updated

Abstract

Depth prediction is at the core of several computer vision applications, such as autonomous driving and robotics. It is often formulated as a regression task in which depth values are estimated through network layers. Unfortunately, the distribution of values on depth maps is seldom explored. Therefore, this paper proposes a novel framework combining contrastive learning and depth prediction, allowing us to pay more attention to depth distribution and consequently enabling improvements to the overall estimation process. Purposely, we propose a window-based contrastive learning module, which partitions the feature maps into non-overlapping windows and constructs contrastive loss within each one. Forming and sorting positive and negative pairs, then enlarging the gap between the two in the representation space, constraints depth distribution to fit the feature of the depth map. Experiments on KITTI and NYU datasets demonstrate the effectiveness of our framework.
BibTeX
@inproceedings{fan2023contrastive,
  author = {Rizhao Fan and Matteo Poggi and Stefano Mattoccia},
  title = {{Contrastive Learning for Depth Prediction}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  url = {https://openreview.net/forum?id=972oAiY59t},
}
Figure