FG-Depth: Flow-Guided Unsupervised Monocular Depth Estimation

Junyu Zhu, Lina Liu, Yong Liu, Wanlong Li, Feng Wen, Hongbo Zhang

Abstract

Unsupervised monocular depth estimation has emerged as a powerful alternative to supervised methods, offering comparable accuracy without the high cost of LiDAR annotations. However, traditional unsupervised frameworks rely heavily on image reconstruction and photometric loss, which are prone to local minima due to lighting variations and textureless regions. In this paper, we present FG-Depth, a framework that breaks this performance bottleneck by guiding the optimization process with prior knowledge from a pretrained Flow-Net. We introduce two primary innovations: (i) a flow distillation loss that replaces the standard photometric loss and (ii) a prior flow-based mask to eliminate noisy pixels during training. Our approach achieves state-of-the-art results on the KITTI and NYU-Depth-v2 datasets, demonstrating superior robustness and generalization.

Introduction

Estimating 3D depth from a single 2D image is a fundamental challenge in computer vision, with applications ranging from autonomous driving to augmented reality. While supervised learning has achieved high precision, the reliance on expensive ground-truth data (like LiDAR) limits its scalability. Unsupervised monocular depth estimation addresses this by using geometric constraints—typically by warping a reference image to a target image and minimizing the difference in pixel intensity, known as photometric loss.

Despite its success, photometric loss has inherent limitations. It assumes Lambertian surfaces (where brightness is independent of the viewing angle) and static scenes. In real-world scenarios, these assumptions are often violated by shadows, reflections, and moving objects. Furthermore, the loss landscape for photometric consistency is often non-convex and riddled with local minima, particularly in low-texture areas like roads or skies. FG-Depth proposes to move beyond these limitations by leveraging the geometric consistency inherent in optical flow.

The FG-Depth Framework

Flow Distillation Loss ($L_{fd}$)

The core innovation of FG-Depth is the shift from color-based supervision to flow-based distillation. Instead of relying solely on raw pixel comparisons, we utilize a pretrained optical flow network (specifically RAFT-Stereo) trained on synthetic data. This network generates "prior flow" fields that act as pseudo-labels for the depth network.

The flow distillation loss ($L_{fd}$) consists of two components:

  1. Depth Regression ($L_{dr}$): Forces the Depth-Net to align its predictions with the pseudo-depth labels derived from the prior flow.
  2. Flow Preservation ($L_{fp}$): Ensures that the reconstructed scene remains consistent with the geometric flow field.

By distilling knowledge from a Flow-Net, the model benefits from a "steeper" and more accurate loss landscape. This allows the optimization process to bypass the local minima that typically entrap photometric-based models.

Prior Flow-based Mask ($M_f$)

Training on real-world datasets like KITTI often involves noisy data, such as objects at extreme distances (e.g., >80 meters) where depth estimation becomes unreliable. FG-Depth introduces a mask based on the magnitude of the prior flow. By identifying and masking out these "out-of-range" or invalid pixels, the framework prevents noisy gradients from corrupting the model’s weights, leading to more stable training and higher accuracy.

Architecture

The system utilizes a ResNet-18 backbone enhanced with Residual U-blocks (RSU). These blocks facilitate better feature bridging across different scales, which is crucial for maintaining high-resolution details in the final depth map.

Experimental Evaluation

Performance on KITTI

We evaluated FG-Depth on the KITTI Eigen split, the standard benchmark for outdoor driving scenes. FG-Depth achieved an Absolute Relative (Abs Rel) error of 0.088, a significant improvement over previous benchmarks such as MonoDepth2 (0.105) and EPCDepth (0.093). Qualitatively, the model produces sharper boundaries for thin structures like traffic signs and poles, and it maintains consistency on smooth road surfaces where other models often produce artifacts.

Cross-Domain Generalization

To test the robustness of the flow-guided approach, we conducted zero-shot testing on the NYU-Depth-v2 indoor dataset. Despite being trained exclusively on outdoor driving data, the FG-Depth model outperformed other unsupervised methods. This suggests that the geometric priors learned from optical flow generalize across domains much more effectively than the appearance-based features learned via photometric loss.

Conclusion

FG-Depth demonstrates that the bottleneck in unsupervised monocular depth estimation is not necessarily the network capacity, but the quality of the supervision signal. By replacing photometric loss with flow distillation and utilizing a prior flow mask, we can achieve state-of-the-art performance that is both more accurate and more robust to environmental variations. Future work will explore extending this framework to monocular video sequences, further reducing the dependency on stereo training data.

Summary updated

Abstract

The great potential of unsupervised monocular depth estimation has been demonstrated by many works due to low annotation cost and impressive accuracy comparable to supervised methods. To further improve the performance, recent works mainly focus on designing more complex network structures and exploiting extra supervised information, e.g., semantic segmentation. These methods optimize the models by exploiting the reconstructed relationship between the target and reference images in varying degrees. However, previous methods prove that this image reconstruction optimization is prone to get trapped in local minima. In this paper, our core idea is to guide the optimization with prior knowledge from pretrained Flow-Net. And we show that the bottleneck of unsupervised monocular depth estimation can be broken with our simple but effective framework named FG-Depth. In particular, we propose (i) a flow distillation loss to replace the typical photometric loss that limits the capacity of the model and (ii) a prior flow based mask to remove invalid pixels that bring the noise in training loss. Extensive experiments demonstrate the effectiveness of each component, and our approach achieves state-of-the-art results on both KITTI and NYU-Depth-v2 datasets.
BibTeX
@inproceedings{zhu2023fg,
  author = {Junyu Zhu and Lina Liu and Yong Liu and Wanlong Li and Feng Wen and Hongbo Zhang},
  title = {{FG-Depth: Flow-Guided Unsupervised Monocular Depth Estimation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160534},
}
Figure