Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences

Vignesh Prasad, Dipanjan Das, Brojeshwar Bhowmick

TL;DR

Researchers from TCS Research have improved how AI learns to see depth from a single moving camera by moving beyond simple "color matching." By forcing the neural network to respect the laws of epipolar geometry and using two consecutive frames instead of one, their model achieves state-of-the-art accuracy in depth and ego-motion estimation while naturally handling tricky moving objects like cars and pedestrians.

What Problem Does This Paper Address?

Estimating depth from a single moving camera (monocular depth estimation) is a "holy grail" for robotics and self-driving cars. Most recent deep learning approaches rely on photometric consistency: they take a video sequence, predict depth and camera motion, and then "warp" one frame onto another. If the warped image matches the original, the model assumes its depth prediction is correct.

However, this approach has a major "fallacy": it assumes the world is static. If a car drives past you, the photometric error will be high even if your depth prediction is perfect, simply because the car moved. Standard models try to fix this with "explainability masks" (essentially the AI guessing which parts of the image to ignore), but these are often unreliable and add complexity.

Methods

The authors propose a more "geometrically sound" pipeline built on two key innovations:

  1. 2-View Depth Inference: Unlike standard models that predict depth from a single static image, this network takes two consecutive frames as input. This allows the network to leverage temporal relationships and "stereo-like" cues, similar to how human vision benefits from motion.
  2. Epipolar Constraint Weighting: This is the "secret sauce." The researchers use Nistér’s Five Point Algorithm to calculate the Essential Matrix (the geometric link between two camera views). They then calculate the epipolar distance for every pixel.
  3. If a pixel (like a point on a building) follows the laws of geometry, it gets a high weight in the training loss.
  4. If a pixel (like a moving car or an occlusion) violates these laws, it gets a low weight.

This effectively "filters" the training process, forcing the AI to focus on geometrically consistent parts of the scene without needing a separate, complex mask-prediction network.

Datasets

The model was trained and evaluated on the KITTI dataset, the industry standard for autonomous driving benchmarks. Specifically:

  • KITTI Raw: Used for training and depth evaluation (using the Eigen split).
  • KITTI Visual Odometry: Used to evaluate how accurately the model tracks the camera's movement (ego-motion).

Key Insights and Ideas

  • Geometry over Heuristics: Instead of letting the network "guess" what to ignore (via explainability masks), the authors use the fundamental laws of computer vision (epipolar lines) to guide the learning.
  • Implicit Motion Handling: By weighting the loss with epipolar distance, the model implicitly learns to handle non-rigid objects. If an object moves independently of the camera, it won't fall on the expected epipolar line, and the model will naturally discount its influence during training.
  • The 2-View Advantage: Moving from 1-view to 2-view depth prediction significantly improves the network's ability to capture inter-pixel relationships, leading to sharper object boundaries and better performance in large, open spaces.

Key Results

The proposed method outperformed several established benchmarks, including the seminal SfMLearner:

  • Depth Accuracy: On the KITTI benchmark, the model achieved an Absolute Relative (Abs Rel) error of 0.166, significantly better than the 0.201-0.208 range of previous purely monocular methods.
  • Ego-Motion: The model showed a lower Average Trajectory Error (ATE) across almost all KITTI sequences compared to SfMLearner.
  • Qualitative Improvements: The depth maps produced are noticeably sharper and more meaningful in "failure case" scenarios, such as texture-less regions or when objects are very close to the camera.

Limitations and Future Work

The authors noted a surprising result: 3-view depth prediction (using three frames instead of two) actually performed worse. They hypothesize that as the "window" of frames grows, the motion becomes too large for the convolutional filters to track effectively, leading to "holes" in the depth map.

Future directions include: * Integrating Semantics: Moving from pixel-level geometry to high-level scene understanding (e.g., knowing that a "car" is an object that can move). * Architectural Coupling: Creating a single, more tightly integrated network that predicts both pose and depth simultaneously to better capture their complex relationship.

Summary updated

Abstract

Deep approaches to predict monocular depth and ego-motion have grown in recent years due to their ability to produce dense depth from monocular images. The main idea behind them is to optimize the photometric consistency over image sequences by warping one view into another, similar to direct visual odometry methods. One major drawback is that these methods infer depth from a single view, which might not effectively capture the relation between pixels. Moreover, simply minimizing the photometric loss does not ensure proper pixel correspondences, which is a key factor for accurate depth and pose estimations. In contrast, we propose a 2-view depth network to infer the scene depth from consecutive frames, thereby learning inter-pixel relationships. To ensure better correspondences, thereby better geometric understanding, we propose incorporating epipolar constraints to make the learning more geometrically sound. We use the Essential matrix obtained using Nistér's Five Point Algorithm, to enforce meaningful geometric constraints, rather than using it as training labels. This allows us to use lesser no. of trainable parameters compared to state-of-the-art methods. The proposed method results in better depth images and pose estimates, which capture the scene structure and motion in a better way. Such a geometrically constrained learning performs successfully even in cases where simply minimizing the photometric error would fail.
BibTeX
@inproceedings{prasad2018epipolar,
  author = {Vignesh Prasad and Dipanjan Das and Brojeshwar Bhowmick},
  title = {{Epipolar Geometry based Learning of Multi-view Depth and Ego-Motion from Monocular Sequences}},
  booktitle = {ICVGIP},
  year = {2018},
  url = {https://openreview.net/forum?id=PI_zd4JzXQa},
}

Loading related content...

Figure