Abstract
Pose estimation and map building are central ingredients of autonomous robots and typically rely on the registration of sensor data. In this paper, we investigate a new metric for registering images that builds upon on the idea of the photometric error. Our approach combines a gradient orientation-based metric with a magnitude-dependent scaling term. We integrate both into stereo estimation as well as visual odometry systems and show clear benefits for typical disparity and direct image registration tasks when using our proposed metric. Our experimental evaluation indicates that our metric leads to more robust and more accurate estimates of the scene depth as well as camera trajectory. Thus, the metric improves camera pose estimation and in turn the mapping capabilities of mobile robots. We believe that a series of existing visual odometry and visual SLAM systems can benefit from the findings reported in this paper.
Introduction
Direct methods for Visual Odometry (VO) and Stereo Matching typically rely on the photometric consistency assumption, which posits that the intensity of a 3D point remains constant across different camera views. However, in real-world robotic applications, this assumption is frequently violated by factors such as automatic exposure adjustments, shadows, lens vignetting, and motion blur. While feature-based methods offer robustness, they are sparse and computationally intensive. This work proposes a dense metric that maintains the simplicity of pixel-wise comparisons while achieving the robustness of complex descriptors.
Scaled Gradient-based Dissimilarity (SGF)
The core contribution of this research is the Scaled Gradient-based Dissimilarity ($e_{sgf}$), which shifts the focus from raw pixel intensities ($I$) to image gradients ($\nabla I$). Gradients capture the orientation and sharpness of edges, which are inherently more stable under varying illumination.
A critical observation in previous gradient-based metrics, such as the Unscaled Gradient-based Dissimilarity ($e_{ugf}$), is their tendency to be "gradient-hungry." Without proper scaling, low-gradient regions often incorrectly match with high-gradient edges because the normalized dot product favors the strongest signal. To address this, the proposed SGF metric introduces a magnitude-dependent scaling term. This ensures that patches are matched based on both orientation and relative contrast, preventing the optimizer from being pulled toward irrelevant sharp edges.
Experimental Integration and Evaluation
The SGF metric was integrated into several state-of-the-art frameworks to demonstrate its "plug-and-play" utility:
- Stereo Matching: Integrated into OpenCV's
StereoBMandMeshStereo. - Visual Odometry: Integrated into
Direct Sparse Odometry (DSO)andBasalt.
Evaluation was conducted across diverse benchmarks, including the synthetic ICL-NUIM (with simulated vignetting), Middlebury Stereo 2014, KITTI Stereo 2015, and the real-world EuRoC MAV dataset.
Results and Discussion
The proposed metric demonstrates superior robustness compared to standard photometric and normalized cross-correlation (NCC) metrics. As shown in the error residual maps, $e_{sgf}$ maintains a consistent, low error profile even when images are subjected to significant lighting modifications and vignetting.
Key performance highlights include:
- Stereo Accuracy: On the Middlebury dataset, SGF achieved a mean disparity error of 3.29 px, significantly outperforming the standard Sum of Absolute Differences (5.80 px).
- VO Robustness: In the EuRoC dataset, SGF-enhanced DSO showed a 10% increase in successful tracking attempts on sequences with high-dynamic-range lighting, where the original photometric-based DSO frequently failed.
- Efficiency: By deriving simplified Jacobians ($e_{sgf2}$, $e_{sgf3}$), the authors ensured the metric remains computationally efficient for real-time robotic applications.
Conclusion
By moving beyond simple photometric consistency to a scaled gradient-based approach, this work significantly enhances the reliability of direct VO and stereo matching. The SGF metric provides a robust alternative for environments with challenging lighting, offering a clear path for improving the mapping and localization capabilities of autonomous mobile robots. Future work may extend this approach to multi-modal registration, such as matching thermal and RGB imagery.
Summary updated