SfMLearner++: Learning Monocular Depth & Ego-Motion Using Meaningful Geometric Constraints

Vignesh Prasad, Brojeshwar Bhowmick

TL;DR

The paper introduces SfMLearner++, an unsupervised deep learning method for monocular depth and ego-motion estimation. By incorporating classical epipolar geometry constraints to weight the photometric loss, the model effectively handles ambiguous pixels and moving objects without needing a dedicated, parameter-heavy "explainability mask." This results in a lighter, more efficient network that achieves state-of-the-art or highly competitive performance across multiple benchmarks.

What Problem Does This Paper Address?

Estimating dense 3D depth and camera motion (ego-motion) from a single moving camera is a notoriously difficult problem. Traditionally, this is tackled by Structure-from-Motion (SfM) or Visual Odometry (VO) algorithms that rely on sparse feature matching.

Recently, unsupervised deep learning methods have shown great promise by optimizing for "photometric consistency"—warping a source image to match a target image based on predicted depth and pose, and minimizing the pixel-wise differences. However, a naive photometric loss struggles with non-rigid scenes (e.g., moving cars, pedestrians) and occlusions, often leading to incorrect pixel correspondences. Previous deep learning solutions addressed this by adding complex sub-networks to predict optical flow or "explainability masks" to ignore moving objects. These additions heavily inflate the number of parameters and computational cost. This paper seeks a simpler, geometrically grounded alternative.

Methods

The authors propose a dual-CNN architecture: a Depth CNN to predict inverse depth from a single image, and a Pose CNN to estimate relative camera motion from a sequence of images.

The key innovations include:

  • Epipolar Constraints: Instead of learning a separate mask to ignore moving objects, the authors leverage classical epipolar geometry. They compute the Essential Matrix ($E$) using Nistér’s Five-Point Algorithm on sparse feature matches.
  • Geometrically Constrained Loss: They use the calculated epipolar distance to weight the standard photometric loss. Specifically, the photometric error of a pixel is multiplied by $e^{|\hat{\tilde{p}}^T E \tilde{p}|}$. Pixels that violate epipolar constraints (like those on moving objects or bad matches) exponentially increase the photometric penalty. This forces the network to implicitly learn geometrically sound projections.
  • Architecture Tweaks: Because the epipolar loss naturally handles geometric inconsistencies, the authors completely remove the "explainability mask" decoder used in the original SfMLearner, significantly reducing the parameter count. They also apply an inverse-depth normalization and an edge-aware spatial smoothness loss to sharpen object boundaries.

Datasets

  • KITTI: Used for both training (Eigen split, ~40K images) and evaluation of depth and pose estimation.
  • Cityscapes: Used for zero-shot evaluation (testing the KITTI-trained model on unseen urban driving data) to demonstrate generalization.
  • Make3D: Used for zero-shot evaluation on non-street, still outdoor images to further test robustness and generalization.

Key Insights and Ideas

  • Classical Geometry Enhances Deep Learning: By injecting a hard geometric constraint (the Essential matrix) into the loss function, the network learns much better correspondences than it would from raw pixel intensities alone.
  • Epipolar Errors as Natural Motion Masks: The epipolar weighting naturally highlights moving objects (like cyclists and cars) because their independent motion violates the static-scene assumption of the camera's ego-motion. This elegantly eliminates the need for a dedicated neural network branch to predict motion masks.
  • Simplicity Rivals Complexity: The method achieves comparable or superior results to highly complex models (like LEGO or GeoNet) that use expensive ICP point-cloud alignments, optical flow decoders, or surface normal consistencies. It proves that well-chosen classical constraints can replace brute-force parameter scaling.

Key Results

  • Depth Estimation (KITTI): Achieved an Absolute Relative (Abs Rel) error of 0.175, outperforming the baseline SfMLearner (0.208) and competing closely with much heavier, state-of-the-art models.
  • Pose Estimation (KITTI): Outperformed both the baseline SfMLearner and the classical Five-Point algorithm in Average Translational Direction Error (ATDE).
  • Ablation: Adding just the epipolar constraint improved the Abs Rel error from 0.221 to 0.217. Combining it with inverse-depth normalization yielded the final 0.175 Abs Rel error, proving both additions are highly effective.
  • Generalization: The KITTI-trained model successfully predicted sharp, coherent depth maps on entirely unseen datasets (Cityscapes and Make3D) without any fine-tuning, demonstrating strong out-of-distribution robustness.

Limitations and Future Work

  • Limitations: The authors note that the current approach only makes pixel-level inferences and lacks a higher-level semantic understanding of the scene.
  • Future Work: They suggest integrating semantic segmentation to better correlate specific object types with depth and ego-motion estimates. They also propose exploring multi-view depth prediction (learning how depth varies across multiple frames) or merging the pose and depth networks into a single, unified architecture to better capture the complex relationship between camera motion and scene structure.

Summary updated

Abstract

Most geometric approaches to monocular Visual Odometry (VO) provide robust pose estimates, but sparse or semi-dense depth estimates. Off late, deep methods have shown good performance in generating dense depths and VO from monocular images by optimizing the photometric consistency between images. Despite being intuitive, a naive photometric loss does not ensure proper pixel correspondences between two views, which is the key factor for accurate depth and relative pose estimations. It is a well known fact that simply minimizing such an error is prone to failures. We propose a method using Epipolar constraints to make the learning more geometrically sound. We use the Essential matrix, obtained using Nistér's Five Point Algorithm, for enforcing meaningful geometric constraints on the loss, rather than using it as labels for training. Our method, although simplistic but more geometrically meaningful, uses lesser number of parameters to give a comparable performance to state-of-the-art methods which use complex losses and large networks showing the effectiveness of using epipolar constraints. Such a geometrically constrained learning method performs successfully even in cases where simply minimizing the photometric error would fail.
BibTeX
@inproceedings{prasad2019sfmlearner,
  author = {Vignesh Prasad and Brojeshwar Bhowmick},
  title = {{SfMLearner++: Learning Monocular Depth \& Ego-Motion Using Meaningful Geometric Constraints}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=sk2Qoav3Ir2},
}

Loading related content...

Figure