SfMLearner++: Learning Monocular Depth & Ego-Motion using Meaningful Geometric Constraints

Vignesh Prasad, Brojeshwar Bhowmick

Abstract

Most geometric approaches to monocular Visual Odometry (VO) provide robust pose estimates, but sparse or semi-dense depth estimates. Off late, deep methods have shown good performance in generating dense depths and VO from monocular images by optimizing the photometric consistency between images. Despite being intuitive, a naive photometric loss does not ensure proper pixel correspondences between two views, which is the key factor for accurate depth and relative pose estimations. It is a well known fact that simply minimizing such an error is prone to failures. We propose a method using Epipolar constraints to make the learning more geometrically sound. We use the Essential matrix, obtained using Nister's Five Point Algorithm, for enforcing meaningful geometric constraints on the loss, rather than using it as labels for training. Our method, although simplistic but more geometrically meaningful, using lesser number of parameters, gives a comparable performance to state-of-the-art methods which use complex losses and large networks showing the effectiveness of using epipolar constraints. Such a geometrically constrained learning method performs successfully even in cases where simply minimizing the photometric error would fail.

Introduction

Estimating dense 3D depth and camera motion (ego-motion) from a single moving camera is a fundamental challenge in computer vision. Traditional Structure-from-Motion (SfM) or Visual Odometry (VO) algorithms typically rely on sparse feature matching. While recent unsupervised deep learning methods have shown promise by optimizing for "photometric consistency"—warping source images to match target images—they often struggle with non-rigid scenes containing moving objects (e.g., cars, pedestrians) and occlusions. These inconsistencies lead to incorrect pixel correspondences. Previous solutions often involved complex sub-networks for optical flow or "explainability masks" to ignore moving objects, significantly increasing computational costs. SfMLearner++ addresses this by introducing a simpler, geometrically grounded alternative.

Methodology

The proposed architecture consists of a dual-CNN setup: a Depth CNN that predicts inverse depth from a single image and a Pose CNN that estimates relative camera motion from a sequence of images.

Overview of the SfMLearner++ architecture, illustrating the flow from input views to predicted depth and pose, and the feedback loop using epipolar-constrained losses.

The core innovation is the integration of Epipolar Constraints. Instead of learning a separate mask to handle moving objects, the method leverages classical epipolar geometry. The Essential Matrix ($E$) is computed using Nistér’s Five-Point Algorithm on sparse feature matches. This matrix is then used to weight the standard photometric loss. Specifically, the photometric error of a pixel is multiplied by $e^{|\hat{\tilde{p}}^T E \tilde{p}|}$. Pixels violating epipolar constraints—typically those on independently moving objects or incorrect matches—receive an exponentially higher penalty. This forces the network to implicitly learn geometrically consistent projections without the need for an explicit explainability mask decoder, thereby reducing the total parameter count.

Experimental Results

The model was trained on the KITTI dataset (Eigen split) and evaluated for both depth and pose estimation. It was also tested in a zero-shot manner on the Cityscapes and Make3D datasets to evaluate generalization.

Depth and Pose Performance

On the KITTI benchmark, SfMLearner++ achieved an Absolute Relative (Abs Rel) error of 0.175, a significant improvement over the baseline SfMLearner (0.208). In terms of pose estimation, the method outperformed both the baseline and the classical Five-Point algorithm in Average Translational Direction Error (ATDE).

Qualitative comparison of depth estimation results on the KITTI dataset. The proposed method produces sharper and more accurate depth maps compared to the baseline SfMLearner.

Generalization and Robustness

Ablation studies confirmed that the epipolar constraint is a key driver of performance, improving the Abs Rel error from 0.221 to 0.217. When combined with inverse-depth normalization, the error dropped to 0.175. Furthermore, the model demonstrated strong robustness by producing coherent depth maps on unseen datasets (Cityscapes and Make3D) without any additional fine-tuning.

Conclusion

SfMLearner++ demonstrates that incorporating well-chosen classical geometric constraints into deep learning frameworks can replace brute-force parameter scaling. By using epipolar errors as natural motion masks, the network achieves state-of-the-art performance with a lighter architecture. Future work will explore the integration of semantic segmentation to improve object-level understanding and multi-view depth prediction to better capture temporal scene structure.

Summary updated

Abstract

Most geometric approaches to monocular Visual Odometry (VO) provide robust pose estimates, but sparse or semi-dense depth estimates. Off late, deep methods have shown good performance in generating dense depths and VO from monocular images by optimizing the photometric consistency between images. Despite being intuitive, a naive photometric loss does not ensure proper pixel correspondences between two views, which is the key factor for accurate depth and relative pose estimations. It is a well known fact that simply minimizing such an error is prone to failures. We propose a method using Epipolar constraints to make the learning more geometrically sound. We use the Essential matrix, obtained using Nister's Five Point Algorithm, for enforcing meaningful geometric constraints on the loss, rather than using it as labels for training. Our method, although simplistic but more geometrically meaningful, using lesser number of parameters, gives a comparable performance to state-of-the-art methods which use complex losses and large networks showing the effectiveness of using epipolar constraints. Such a geometrically constrained learning method performs successfully even in cases where simply minimizing the photometric error would fail.
BibTeX
@inproceedings{prasad2019sfmlearner,
  author = {Vignesh Prasad and Brojeshwar Bhowmick},
  title = {{SfMLearner++: Learning Monocular Depth \& Ego-Motion using Meaningful Geometric Constraints}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=sk2Qoav3Ir2},
}
Figure