Learning-Based Image Enhancement for Visual Odometry in Challenging HDR Environments

Ruben Gomez-Ojeda, Zichao Zhang, Javier González Jiménez, Davide Scaramuzza

Abstract

Visual Odometry (VO) algorithms, essential for robotic navigation, frequently fail in High Dynamic Range (HDR) environments where rapid changes in illumination violate the fundamental assumption of brightness constancy. This paper proposes a learning-based preprocessing pipeline designed to transform challenging images—whether over-exposed, under-exposed, or subject to flickering—into informative representations optimized for VO tracking. By employing a combination of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) units, the framework ensures both high gradient saliency and temporal consistency. Experimental results using ORB-SLAM and DSO show that this approach significantly enhances tracking robustness and accuracy in extreme lighting conditions where standard methods fail.

Introduction: The HDR Challenge in Visual Odometry

Visual Odometry (VO) estimates a robot's ego-motion by tracking interest points across successive camera frames. This process relies on two critical factors: the presence of distinct image gradients and the assumption of brightness constancy—the premise that a point's intensity remains stable between frames.

In real-world HDR environments, such as a vehicle exiting a dark tunnel into bright sunlight, these assumptions are often violated. Standard camera sensors typically lose information in deep shadows or blown-out highlights. Furthermore, auto-exposure algorithms can cause sudden, global brightness shifts that break the data association required for VO. While traditional approaches have attempted to model camera response functions or manually tune exposure, they often lack the flexibility to generalize across diverse sensors and dynamic environments.

Proposed Architectures

To address these challenges, the authors developed two distinct deep learning architectures tailored for different computational constraints:

  1. FT-LSTM (Fine-Tuned CNN + LSTM): This high-performance model utilizes the first eight layers of a pre-trained VGGNet as an encoder. The core innovation is the inclusion of LSTM layers, which leverage temporal information from previous frames to ensure the enhanced output sequence is stable and free from flickering.
  2. Small-CNN: A lightweight, three-layer encoder-decoder network designed for real-time robotic applications. While less complex than the FT-LSTM, it provides a significant boost to VO performance with minimal latency.

Training Methodology

A primary challenge in training such a network is the lack of "ground truth" for what constitutes an ideal VO image. The authors addressed this through a multi-stage training strategy:

  • Logarithmic Loss: Instead of standard Euclidean distance, the models were pre-trained using Logarithmic Root Mean Square Error (LogRMSE). This loss function handles large intensity errors more gracefully, leading to smoother image reconstructions and faster convergence.
  • Siamese Invariance: Using a Siamese network configuration, the model was forced to produce identical outputs for the same scene regardless of whether the input was dark, bright, or over-exposed. This "learned invariance" is key to maintaining tracking through illumination changes.
  • Temporal Consistency: The LSTM components were trained specifically on image sequences to prevent the "enhanced" brightness from jumping between frames, thereby preserving the brightness constancy assumption for the downstream VO algorithm.

Experimental Results

The proposed enhancement pipeline was evaluated against standard image processing techniques, such as Histogram Equalization, using state-of-the-art VO systems like ORB-SLAM and DSO.

Robustness and Accuracy

In "extreme" test scenarios involving flickering lights and high-contrast transitions, standard VO algorithms frequently lost tracking. In contrast, images preprocessed by FT-LSTM and Small-CNN allowed these algorithms to maintain continuous tracking with significantly lower Root Mean Square Error (RMSE). For instance, in a "bright-switch" test, the FT-CNN reduced ORB-SLAM’s trajectory error from 3.13% to 1.97%.

Real-time Performance

Computational efficiency is critical for robotics. The Small-CNN demonstrated impressive speed, processing a 756x480 frame in approximately 48.4 ms on a GTX Titan. While the more accurate FT-LSTM required 275 ms per frame—making it less suitable for high-speed flight—the Small-CNN offers a viable path for real-time integration on mobile platforms.

Discussion and Conclusion

The core insight of this work is that VO does not require aesthetically pleasing images; it requires images with maximized gradient information. By acting as "learned gradient boosters," these networks recover structural details from underexposed regions that are otherwise invisible to standard algorithms.

While the models were trained primarily on synthetic data (e.g., the New University of Tsukuba and Urban Virtual datasets), they generalized remarkably well to real-world sequences. This suggests that the underlying "physics" of light distortion captured by the model is universal. Future work will focus on reducing the computational latency of the high-accuracy models and exploring unsupervised learning techniques to further bridge the gap between synthetic training and real-world deployment.

Summary updated

BibTeX
@inproceedings{gomezojeda2018learning,
  author = {Ruben Gomez-Ojeda and Zichao Zhang and Javier González Jiménez and Davide Scaramuzza},
  title = {{Learning-Based Image Enhancement for Visual Odometry in Challenging HDR Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  doi = {10.1109/icra.2018.8462876},
}
Figure