Beyond Photometric Consistency: Gradient-based Dissimilarity for Improving Visual Odometry and Stereo Matching

Jan Quenzel, Radu Alexandru Rosu, Thomas Läbe, Cyrill Stachniss, Sven Behnke

This ICRA 2020 paper from Jan Quenzel and colleagues at the University of Bonn addresses a fundamental challenge in computer vision: how to make robots "see" and track their movement reliably when lighting conditions are far from perfect.

TL;DR

The authors introduce a novel Scaled Gradient-based Dissimilarity (SGF) metric that replaces standard pixel-intensity comparisons in visual odometry and stereo matching. By focusing on the orientation and scaled magnitude of image gradients rather than raw brightness, their approach significantly improves tracking robustness and depth estimation in environments with changing exposure and lens vignetting.

What Problem Does This Paper Address?

Most "direct" methods for Visual Odometry (VO) and Stereo Matching rely on the photometric consistency assumption: the idea that a point in the world should have the same pixel intensity across different camera views.

In the real world, this assumption is fragile. Automatic exposure adjustments, shadows, lens vignetting (darkening at the corners), and motion blur all cause pixel values to change even if the underlying scene hasn't. While "feature-based" methods (like SIFT or ORB) are more robust to these changes, they are sparse and computationally expensive. The authors sought a "middle ground"—a dense metric that is as easy to compute as pixel differences but as robust as complex descriptors.

Methods

The core contribution is the Scaled Gradient-based Dissimilarity ($e_{sgf}$).

  1. Beyond Intensity: Instead of comparing raw brightness ($I$), the metric compares image gradients ($\nabla I$). Gradients represent the direction and "sharpness" of edges, which are much more stable under lighting changes.
  2. Orientation vs. Magnitude: Previous work used the dot product of normalized gradients to match orientations. However, the authors found a flaw: low-gradient (flat) regions would often incorrectly "latch onto" high-gradient (sharp) edges because the math favored the strongest signal.
  3. The "Scaling" Secret Sauce: The authors introduced a magnitude-dependent scaling term. This ensures that a low-contrast patch only matches another low-contrast patch with a similar orientation, preventing the "gradient-hungry" behavior of previous metrics.
  4. Integration: They didn't just build a theoretical model; they integrated this metric into four major existing systems:
  5. OpenCV StereoBM and MeshStereo (for depth from two cameras).
  6. DSO (Direct Sparse Odometry) and Basalt (for tracking camera motion).

Datasets

The researchers evaluated their metric across several gold-standard benchmarks:

  • ICL-NUIM: A synthetic dataset where they manually injected vignetting and exposure changes to test controlled robustness.
  • Middlebury Stereo 2014 & KITTI Stereo 2015: Standard benchmarks for evaluating how accurately a system can calculate depth (disparity) from stereo pairs.
  • EuRoC MAV Dataset: Real-world data from a micro-aerial vehicle flying in industrial environments, used to test trajectory accuracy (Absolute Trajectory Error).

Key Insights and Ideas

  • Orientation is King: The direction of an edge is a much more reliable "fingerprint" for a pixel than its actual brightness value.
  • Magnitude Matters (Relatively): You can't just normalize gradients and throw away the magnitude. Doing so makes the system hypersensitive to noise in flat areas (like a white wall). The SGF metric balances orientation and magnitude to keep the optimizer on track.
  • Plug-and-Play Robustness: One of the most powerful insights is that you don't need to reinvent the SLAM pipeline. By simply swapping the "error function" (the way the computer calculates how "wrong" a match is), you can gain significant performance boosts in existing state-of-the-art frameworks like DSO.

Key Results

  • Stereo Accuracy: On the Middlebury dataset, the proposed SGF metric achieved a mean disparity error of 3.29 px, significantly outperforming the standard Sum of Absolute Differences (5.80 px) and even sophisticated PatchMatch techniques (4.56 px).
  • VO Robustness: When integrated into DSO and tested on the EuRoC dataset, the SGF metric reduced the Absolute Trajectory Error (ATE). Most notably, it increased successful tracking attempts by 10% on sequences with high-dynamic-range lighting where the original DSO often failed.
  • Efficiency: The authors derived simplified versions of the metric ($e_{sgf2}$ and $e_{sgf3}$) with easier-to-calculate Jacobians, allowing the system to run at the real-time frame rates required for robotics.

Limitations and Future Work

While the metric is robust, the authors note that the Jacobian calculation (the math used to update the camera's position) is more complex than simple photometric differences. This can make the optimization process slightly more difficult to converge if the initial guess is poor.

Future work could involve applying this gradient-based approach to multi-modal registration—for example, matching a thermal image to a standard RGB image, where intensities differ wildly but edge orientations often remain consistent.

Summary updated

Abstract

Pose estimation and map building are central ingredients of autonomous robots and typically rely on the registration of sensor data. In this paper, we investigate a new metric for registering images that builds upon on the idea of the photometric error. Our approach combines a gradient orientation-based metric with a magnitude-dependent scaling term. We integrate both into stereo estimation as well as visual odometry systems and show clear benefits for typical disparity and direct image registration tasks when using our proposed metric. Our experimental evaluation indicats that our metric leads to more robust and more accurate estimates of the scene depth as well as camera trajectory. Thus, the metric improves camera pose estimation and in turn the mapping capabilities of mobile robots. We believe that a series of existing visual odometry and visual SLAM systems can benefit from the findings reported in this paper.
BibTeX
@inproceedings{quenzel2020beyond,
  author = {Jan Quenzel and Radu Alexandru Rosu and Thomas Läbe and Cyrill Stachniss and Sven Behnke},
  title = {{Beyond Photometric Consistency: Gradient-based Dissimilarity for Improving Visual Odometry and Stereo Matching}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=HL5dfmj_g06},
}

Loading related content...

Figure