Embodiment: Self-Supervised Depth Estimation Based on Camera Models

Jinchang Zhang, Praveen Kumar Reddy, Xue-Iuan Wong, Yiannis Aloimonos, Guoyu Lu

Abstract: Monocular depth estimation often suffers from scale ambiguity, making it difficult for models to distinguish between small nearby objects and large distant ones. This paper introduces "Embodiment Physics Depth," a framework that leverages known camera geometry and semantic segmentation to provide absolute scale supervision. By calculating the distance to the ground plane and objects touching it, the model achieves state-of-the-art performance on benchmarks like KITTI without requiring LiDAR data for training.

Introduction

Monocular depth estimation—predicting 3D structure from a single 2D image—is inherently an "ill-posed" problem due to scale ambiguity. Current solutions typically fall into two categories: supervised learning, which requires expensive LiDAR data, and self-supervised learning, which often struggles with "scale drift" and produces only relative depth.

The authors address this by treating the camera as an "embodied" agent. By utilizing the physical properties of the camera—specifically its mounting height and lens intrinsics—they mathematically determine the absolute distance to any point on the ground, providing a source of "free" supervision that anchors the scene in real-world meters.

Methodology: The Embodiment Physics Depth Framework

The proposed framework bridges the gap between supervised and self-supervised learning by treating the camera as a physical object with known geometry. The system operates through a multi-stage pipeline that integrates geometric priors with neural network training.

Overview of the Embodiment Physics Depth training pipeline, showing the integration of camera models, semantic segmentation, and multi-phase losses.

1. Physics Depth Calculation

The core innovation lies in calculating "physics depth" using a pinhole camera model. By knowing the camera's height ($h$) and its intrinsic parameters, the distance to any pixel identified as "ground" or "road" via semantic segmentation can be determined through simple trigonometry. This provides a dense, geometrically accurate depth map for the most reliable reference plane in outdoor scenes.

Visualization of the physics depth generation process, from RGB image and semantic masks to calculated depth maps for road and ground surfaces.

2. Edge Extension and Inpainting

To extend depth information beyond the road surface, the authors employ "Edge Extended Physics Depth." They assume that objects touching the ground (such as vehicles or buildings) share the same depth as the ground at their base. This value is propagated upwards, and Telea Inpainting is used to fill gaps, resulting in a dense depth map suitable for supervision.

3. Two-Phase Training

  • Phase 1 (Supervised): The network is initially trained using the calculated physics depth as a ground-truth label. This establishes an absolute scale from the start.
  • Phase 2 (Self-Supervised): The model is fine-tuned using standard photometric loss and a novel 2D Spatial Consistency Loss. This loss uses optical flow to ensure the predicted 3D motion of the scene aligns with the observed 2D movement of pixels between video frames.

Experimental Results

The model was evaluated on the KITTI and Cityscapes datasets, with "zero-shot" testing performed on Make3D to assess generalization. Using SQLDepth as a backbone, the proposed method outperformed existing leaders like Monodepth2 and MonoViT across key metrics (AbsRel, RMSE).

Key findings include:

  • High Precision: The physics depth logic alone proved remarkably accurate, with over 99% of road pixels showing less than 10% error compared to LiDAR ground truth.
  • Absolute Scale: Unlike traditional self-supervised methods, this approach produces depth in meters directly, eliminating the need for post-hoc scaling factors.
  • Generalization: In zero-shot tests on Make3D, the model maintained superior sharpness and geometric detail, indicating it learned fundamental real-world geometry rather than dataset-specific patterns.

Conclusion and Future Work

By "embedding" the physical properties of the camera into the learning process, the authors demonstrate that absolute scale can be recovered without expensive sensors. The primary limitation remains the "Flat World" assumption; the current physics depth calculation assumes a level ground plane. Future research may focus on integrating dynamic camera pitch/roll adjustments from IMU data to handle steep hills and non-flat terrain more robustly.

Summary updated

BibTeX
@inproceedings{zhang2024embodiment,
  author = {Jinchang Zhang and Praveen Kumar Reddy and Xue-Iuan Wong and Yiannis Aloimonos and Guoyu Lu},
  title = {{Embodiment: Self-Supervised Depth Estimation Based on Camera Models}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801343},
}
Figure