UnVELO: Unsupervised Vision-Enhanced LiDAR Odometry with Online Correction

Bin Li, Haifeng Ye, Sihan Fu, Xiaojin Gong, Zhiyu XiangSensors 2023 · DOI

TL;DR

This paper introduces UnVELO, a novel unsupervised visual-LiDAR odometry system that leverages a LiDAR-dominant fusion approach. By converting 3D LiDAR points into dense vertex maps and colorizing them with visual information, UnVELO achieves robust pose estimation without relying on noisy depth predictions, and further refines its accuracy through online pose correction.

What Problem Does This Paper Address?

Accurate and robust estimation of an agent's motion (odometry) is crucial for autonomous systems. While visual odometry (VO) and LiDAR odometry (LO) have seen significant advancements, fusing these two modalities for odometry (visual-LiDAR odometry or VLO) remains an under-explored area, especially within the realm of unsupervised learning. Existing learning-based VLOs often adopt a "vision-dominant" approach, projecting sparse LiDAR data into the camera's view. This leads to sparse depth maps, which are challenging to work with and can introduce noise, limiting overall accuracy. Furthermore, many methods rely on view synthesis losses, which are sensitive to illumination changes. The authors aim to bridge this gap by proposing a LiDAR-dominant VLO that is more robust to illumination variations and avoids the pitfalls of noisy depth predictions.

Methods

UnVELO employs a LiDAR-dominant fusion strategy. The core steps are:

  • Data Pre-processing:
    • Vertex Map Generation: 3D LiDAR point clouds are projected onto a 2D plane using spherical projection, creating a dense vertex map. This is a standard technique in LO.
  • Normal Map Estimation: Surface normals are computed for each point using Singular Value Decomposition (SVD) on local neighborhoods. A confidence map is derived from normal similarity, indicating planar regions.
  • Vertex Color Map Generation: Visual information from synchronized camera images is used to colorize the vertices in the vertex map, creating a vertex color map. This step fuses visual and LiDAR data.
  • Pose Estimation Network: A fully convolutional neural network takes the concatenated vertex maps and vertex color maps as input. It's trained using a combined loss function:
  • Geometric Loss (Lgeo): Applied to locally planar regions with high normal confidence. It uses a point-to-plane distance metric to measure registration error between consecutive LiDAR frames.
  • Visual Loss (Lvis): Applied to textured and cluttered regions with low normal confidence. It measures photometric error between corresponding pixels in the vertex color maps of consecutive frames. This loss helps maintain visual consistency.
  • Online Pose Correction (OPC): During inference (test time), the pose predicted by the trained network is further refined. This module uses the same unsupervised loss functions as training, optimizing the pose iteratively without requiring ground truth labels. Hard sample mining is incorporated to focus the optimization on more challenging correspondences, improving convergence and stability.

Datasets

The proposed method was evaluated on two datasets:

  • KITTI Odometry Benchmark: A widely used dataset for autonomous driving research, providing synchronized LiDAR scans and camera images.
  • DSEC (Driving Stereo Event Camera) Dataset: This dataset features synchronized stereo RGB cameras and LiDAR. Notably, the LiDAR in DSEC has only 16 lines, making it sparse, and the camera and LiDAR are not perfectly synchronized. For VLO evaluation, 3D point clouds were derived from disparity maps.

Key Insights and Ideas

  • LiDAR-Dominant Fusion Advantage: By prioritizing LiDAR data for geometric structure and enhancing it with visual color information, the method avoids the need for depth prediction networks, which are often a source of noise and inaccuracies in vision-dominant VLOs.
  • Complementary Losses: The use of both a geometric loss (for reliable planar structures) and a visual loss (for textured, cluttered regions) effectively exploits the complementary nature of LiDAR and camera data, leading to improved robustness, especially under varying illumination conditions.
  • Online Pose Correction Effectiveness: The unsupervised nature of the loss functions allows for effective online pose refinement at test time. The LiDAR-dominant approach makes this correction more efficient and impactful compared to vision-dominant methods.
  • Hard Sample Mining for Optimization: Incorporating hard sample mining into the online correction process significantly improves convergence speed and stability, leading to better final pose estimates.

Key Results

  • KITTI Performance: UnVELO3 (with both geometric and visual losses) outperformed previous two-frame-based learning methods on KITTI sequences 09-10. It was also competitive with hybrid methods that incorporate global optimization across multiple frames.
  • DSEC Performance: On the challenging DSEC dataset, UnVELO demonstrated superior performance, especially in night-time sequences with significant illumination variations, achieving a translation error of 5.84% and a rotation error of 7.92 deg/100m.
  • Online Correction Benefits: The online correction module consistently improved the performance of UnVELO, with optimal results achieved around 40 iterations.
  • Efficiency: The LiDAR-dominant approach of UnVELO was observed to be more efficient than vision-dominant counterparts, particularly when online correction was applied.

Limitations and Future Work

  • Limitations: The paper notes that the generated vertex map can become sparser under poor illumination conditions, especially when derived from disparity maps (as in DSEC). The LiDAR used in DSEC is also very sparse.
  • Future Work: The authors suggest exploring long-term temporal constraints for pose correction to enhance robustness against abrupt motion changes, dynamic objects, and other disturbances.

Summary updated

BibTeX
@article{li2023unvelo,
  author = {Bin Li and Haifeng Ye and Sihan Fu and Xiaojin Gong and Zhiyu Xiang},
  title = {{UnVELO: Unsupervised Vision-Enhanced LiDAR Odometry with Online Correction}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23083967},
}
Figure