Stereo Visual Odometry Pose Correction through Unsupervised Deep Learning

Sumin Zhang, Shouyi Lu, Rui He, Zhipeng Bao2021 · DOI

TL;DR

This paper introduces an unsupervised deep learning network that corrects the pose estimations of traditional stereo visual odometry (VO) systems. By learning from unlabeled stereo image sequences, their method significantly improves accuracy without requiring ground truth data, bringing classical VO systems closer to state-of-the-art performance.

What Problem Does This Paper Address?

Visual Simultaneous Localization and Mapping (VSLAM) is crucial for robot navigation. At its core, Visual Odometry (VO) estimates a robot's movement using camera images. Traditional VO methods, while geometrically sound, rely on assumptions about the environment (e.g., constant illumination, static scenes) that often fail in real-world, challenging conditions. While deep learning (DL) offers robustness, purely DL-based VO often lacks the interpretability and geometric grounding of classical methods, and typically requires large amounts of labeled data for training. This paper aims to bridge this gap by combining the strengths of both approaches: the geometric rigor of classical stereo VO and the robustness and data efficiency of unsupervised deep learning.

Methods

The authors propose an "unsupervised stereo visual odometry pose correction network." This network takes the output pose from a classical stereo VO system (like libviso2-s, DSO, or ORB-SLAM2) and a sequence of stereo images as input. Instead of replacing the classical VO entirely, it learns to predict a "pose correction" that refines the classical VO's output.

Key aspects of their method include:

  • Unsupervised Learning: The network is trained without ground truth pose data. It leverages geometric consistency losses derived from the stereo image sequences themselves.
  • Hybrid Approach: It integrates a classical VO system with a deep neural network. The network learns to correct the output of the classical system, rather than replacing its core functionality.
  • Network Architecture: A modified U-Net encoder-decoder architecture is used. This architecture is designed to process stereo image pairs and output three things simultaneously:
  • Pose Correction: A transformation that refines the classical VO's pose estimate.
  • Depth Map: An estimate of the scene's depth.
  • Explainability Mask: A mask that indicates which pixels are reliable for geometric consistency, helping to filter out dynamic objects or unreliable features.
  • Loss Functions: The training relies on a combination of spatial and temporal geometric consistency losses. This includes:
  • Photometric Consistency Loss: Ensures that pixels in corresponding stereo images have similar intensity values after warping.
  • Disparity Consistency Loss: Enforces consistency between depth estimates and disparity maps.
  • Temporal Photometric Consistency Loss: Ensures consistency across consecutive frames.
  • Explainability Mask Loss: Regularizes the explainability mask to be more robust.

Datasets

The paper uses the KITTI dataset for training and evaluation. Specifically, they train on sequences 00, 02, 05-10 from the "city," "residential," and "road" categories, which contain approximately 46,000 training pairs. These sequences are noted for including challenging scenarios like moving objects, uneven illumination, and occlusions.

Key Insights and Ideas

  • Unsupervised Correction is Powerful: The core insight is that a deep network can learn to correct existing VO estimates effectively, even without ground truth. This is more efficient than training a full VO system from scratch with supervised data.
  • Leveraging Classical Geometry: By keeping the classical VO's geometric constraints and only learning a correction, the method retains interpretability and robustness, unlike purely end-to-end DL approaches.
  • Multi-task Learning Benefits: Simultaneously predicting pose correction, depth, and an explainability mask seems to synergize. The depth and mask likely provide crucial geometric cues that aid in learning a more accurate pose correction.
  • Robustness to Challenging Scenes: Training on challenging KITTI scenes (moving objects, poor lighting) leads to a more robust pose correction network.

Key Results

The proposed method significantly improves the accuracy of the classical stereo VO system (libviso2-s).

  • Quantitative Improvements:
    • Average Absolute Trajectory Error (c-ATE) on KITTI sequences 00, 02, 05-10: 13.77 cm.
  • Average translational relative pose error: 0.038 m.
  • Average translational root-mean-square drift over 100-800m segments: 1.08%.
  • Qualitative Improvements: Trajectory plots show the corrected trajectories closely matching the ground truth, with substantial improvements over the original libviso2-s trajectories, especially in sequence 00.
  • Comparison: The corrected system achieves accuracy comparable to state-of-the-art methods like ORB-SLAM2, while maintaining real-time performance (42.73 ms per pose estimation for the corrected libviso2-s).

Limitations and Future Work

  • Map Points Not Optimized: The current network corrects camera pose but does not optimize map points. The authors suggest extending the system to a full visual SLAM system that optimizes map points in the future.
  • Potential for Further Improvement: While the corrected system is highly accurate, they believe matching with state-of-the-art stereo VO systems could yield even better results.
  • Sensor Fusion: They plan to incorporate other sensor data, such as inertial measurement unit (IMU) data, to further enhance translation corrections.

Summary updated

BibTeX
@misc{zhang2021stereo,
  author = {Sumin Zhang and Shouyi Lu and Rui He and Zhipeng Bao},
  title = {{Stereo Visual Odometry Pose Correction through Unsupervised Deep Learning}},
  year = {2021},
  doi = {10.3390/s21144735},
}
Figure