Abstract
Most geometric approaches to monocular Visual Odometry (VO) provide robust pose estimates, but sparse or semi-dense depth estimates. Off late, deep methods have shown good performance in generating dense depths and VO from monocular images by optimizing the photometric consistency between images. Despite being intuitive, a naive photometric loss does not ensure proper pixel correspondences between two views, which is the key factor for accurate depth and relative pose estimations. It is a well known fact that simply minimizing such an error is prone to failures. We propose a method using Epipolar constraints to make the learning more geometrically sound. We use the Essential matrix, obtained using Nister's Five Point Algorithm, for enforcing meaningful geometric constraints on the loss, rather than using it as labels for training. Our method, although simplistic but more geometrically meaningful, using lesser number of parameters, gives a comparable performance to state-of-the-art methods which use complex losses and large networks showing the effectiveness of using epipolar constraints. Such a geometrically constrained learning method performs successfully even in cases where simply minimizing the photometric error would fail.
Introduction
Estimating dense 3D depth and camera motion (ego-motion) from a single moving camera is a fundamental challenge in computer vision. Traditional Structure-from-Motion (SfM) or Visual Odometry (VO) algorithms typically rely on sparse feature matching. While recent unsupervised deep learning methods have shown promise by optimizing for "photometric consistency"—warping source images to match target images—they often struggle with non-rigid scenes containing moving objects (e.g., cars, pedestrians) and occlusions. These inconsistencies lead to incorrect pixel correspondences. Previous solutions often involved complex sub-networks for optical flow or "explainability masks" to ignore moving objects, significantly increasing computational costs. SfMLearner++ addresses this by introducing a simpler, geometrically grounded alternative.
Methodology
The proposed architecture consists of a dual-CNN setup: a Depth CNN that predicts inverse depth from a single image and a Pose CNN that estimates relative camera motion from a sequence of images.
The core innovation is the integration of Epipolar Constraints. Instead of learning a separate mask to handle moving objects, the method leverages classical epipolar geometry. The Essential Matrix ($E$) is computed using Nistér’s Five-Point Algorithm on sparse feature matches. This matrix is then used to weight the standard photometric loss. Specifically, the photometric error of a pixel is multiplied by $e^{|\hat{\tilde{p}}^T E \tilde{p}|}$. Pixels violating epipolar constraints—typically those on independently moving objects or incorrect matches—receive an exponentially higher penalty. This forces the network to implicitly learn geometrically consistent projections without the need for an explicit explainability mask decoder, thereby reducing the total parameter count.
Experimental Results
The model was trained on the KITTI dataset (Eigen split) and evaluated for both depth and pose estimation. It was also tested in a zero-shot manner on the Cityscapes and Make3D datasets to evaluate generalization.
Depth and Pose Performance
On the KITTI benchmark, SfMLearner++ achieved an Absolute Relative (Abs Rel) error of 0.175, a significant improvement over the baseline SfMLearner (0.208). In terms of pose estimation, the method outperformed both the baseline and the classical Five-Point algorithm in Average Translational Direction Error (ATDE).
Generalization and Robustness
Ablation studies confirmed that the epipolar constraint is a key driver of performance, improving the Abs Rel error from 0.221 to 0.217. When combined with inverse-depth normalization, the error dropped to 0.175. Furthermore, the model demonstrated strong robustness by producing coherent depth maps on unseen datasets (Cityscapes and Make3D) without any additional fine-tuning.
Conclusion
SfMLearner++ demonstrates that incorporating well-chosen classical geometric constraints into deep learning frameworks can replace brute-force parameter scaling. By using epipolar errors as natural motion masks, the network achieves state-of-the-art performance with a lighter architecture. Future work will explore the integration of semantic segmentation to improve object-level understanding and multi-view depth prediction to better capture temporal scene structure.
Summary updated