Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

Vignesh Prasad, Dipanjan Das, Brojeshwar Bhowmick

Abstract

Deep approaches to predict monocular depth and ego-motion have grown in recent years due to their ability to produce dense depth from monocular images. The main idea behind them is to optimize the photometric consistency over image sequences by warping one view into another, similar to direct visual odometry methods. One major drawback is that these methods infer depth from a single view, which might not effectively capture the relation between pixels. Moreover, simply minimizing the photometric loss does not ensure proper pixel correspondences, which is a key factor for accurate depth and pose estimations. In contrast, we propose a 2-view depth network to infer the scene depth from consecutive frames, thereby learning inter-pixel relationships. To ensure better correspondences, thereby better geometric understanding, we propose incorporating epipolar constraints to make the learning more geometrically sound. We use the Essential matrix obtained using Nist'er's Five Point Algorithm, to enforce meaningful geometric constraints, rather than using it as training labels. This allows us to use lesser no. of trainable parameters compared to state-of-the-art methods. The proposed method results in better depth images and pose estimates, which capture the scene structure and motion in a better way. Such a geometrically constrained learning performs successfully even in cases where simply minimizing the photometric error would fail.

Introduction

Estimating depth and ego-motion from monocular video sequences is a critical challenge for autonomous navigation. Most contemporary self-supervised frameworks rely on the principle of photometric consistency: predicting depth and camera motion to reconstruct one frame from another. However, this approach inherently assumes a static world. In real-world scenarios, moving objects (like cars or pedestrians) create high photometric errors that can mislead the training process. While many methods attempt to mitigate this using "explainability masks" to ignore dynamic regions, these masks are often learned heuristically and add significant model complexity.

Methodology

Our approach introduces two significant innovations to ground the learning process in geometric reality:

2-View Depth Inference

Unlike standard models that predict depth from a single static image, our network utilizes two consecutive frames as input. This architectural choice allows the model to leverage temporal relationships and motion-based depth cues, effectively learning inter-pixel relationships that are missed in single-view approaches.

Epipolar Constraint Weighting

To ensure geometrically sound learning, we incorporate explicit epipolar constraints. Using Nistér’s Five Point Algorithm, we compute the Essential Matrix between camera views. We then calculate the epipolar distance for every pixel. This distance serves as a weighting factor in the loss function:

  • Geometrically Consistent Pixels: Points that follow the laws of epipolar geometry (e.g., static background) receive high weights.
  • Violating Pixels: Points that deviate from the expected epipolar lines (e.g., independently moving objects or occlusions) are down-weighted.

This mechanism allows the network to implicitly handle non-rigid motion and focus on stable scene structures without requiring a separate mask-prediction network.

The proposed pipeline integrates a 2-view Depth CNN and a Pose CNN, using epipolar error images to weight the training losses and gradients.

Experimental Results

The proposed method was rigorously evaluated on the KITTI dataset, focusing on both depth accuracy and visual odometry.

Depth Estimation

On the KITTI Raw dataset (Eigen split), our model achieved an Absolute Relative (Abs Rel) error of 0.166. This represents a substantial improvement over baseline monocular methods like SfMLearner, which typically range between 0.201 and 0.208. Qualitatively, the resulting depth maps are sharper and more resilient to common failure modes like texture-less regions or near-field objects.

Qualitative comparison on the KITTI dataset. The proposed method produces depth maps with superior structural clarity and fewer artifacts compared to SfMLearner.

Ego-Motion

Evaluation on the KITTI Visual Odometry benchmark showed that our model consistently achieves a lower Average Trajectory Error (ATE) across most sequences compared to SfMLearner, demonstrating that better depth leads to more accurate pose estimation.

Discussion and Conclusion

By prioritizing fundamental geometric laws over heuristic masking, we demonstrate that monocular depth estimation can be made more robust and accurate. Interestingly, our experiments showed that extending the input to 3-view depth prediction actually degraded performance, likely due to large motions exceeding the receptive fields of standard convolutional filters.

Future work will focus on integrating semantic scene understanding to further distinguish between static and dynamic objects and exploring tighter architectural coupling between the depth and pose networks.

Summary updated

Abstract

Deep approaches to predict monocular depth and ego-motion have grown in recent years due to their ability to produce dense depth from monocular images. The main idea behind them is to optimize the photometric consistency over image sequences by warping one view into another, similar to direct visual odometry methods. One major drawback is that these methods infer depth from a single view, which might not effectively capture the relation between pixels. Moreover, simply minimizing the photometric loss does not ensure proper pixel correspondences, which is a key factor for accurate depth and pose estimations. In contrast, we propose a 2-view depth network to infer the scene depth from consecutive frames, thereby learning inter-pixel relationships. To ensure better correspondences, thereby better geometric understanding, we propose incorporating epipolar constraints to make the learning more geometrically sound. We use the Essential matrix obtained using Nist'er's Five Point Algorithm, to enforce meaningful geometric constraints, rather than using it as training labels. This allows us to use lesser no. of trainable parameters compared to state-of-the-art methods. The proposed method results in better depth images and pose estimates, which capture the scene structure and motion in a better way. Such a geometrically constrained learning performs successfully even in cases where simply minimizing the photometric error would fail.
BibTeX
@inproceedings{prasad2018epipolar,
  author = {Vignesh Prasad and Dipanjan Das and Brojeshwar Bhowmick},
  title = {{Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences}},
  booktitle = {ICVGIP},
  year = {2018},
  url = {https://openreview.net/forum?id=PI_zd4JzXQa},
}
Figure