LodoNet: A Deep Neural Network with 2D Keypoint Matchingfor 3D LiDAR Odometry Estimation

Ce Zheng, Yecheng Lyu, Ming Li, Ziming Zhang

TL;DR

This paper introduces LodoNet, a novel deep learning approach for 3D LiDAR odometry that leverages 2D image feature extraction techniques. By projecting LiDAR point clouds into spherical images and applying SIFT for keypoint matching, LodoNet effectively estimates vehicle motion, achieving state-of-the-art or comparable results on the KITTI benchmark.

What Problem Does This Paper Address?

Estimating a vehicle's precise position and orientation (odometry) is crucial for autonomous driving and robotics. While various sensors like cameras and IMUs can be used, LiDAR offers accurate distance measurements and is less affected by lighting conditions, making it a preferred choice for robust odometry. Traditional LiDAR odometry methods often rely on point cloud registration (like ICP) or global feature matching, which can be computationally expensive and struggle with capturing fine local details. Existing deep learning approaches for LiDAR odometry typically process raw 3D point clouds, which is also computationally demanding. This paper addresses the challenge of developing an accurate, efficient, and robust LiDAR odometry system by bridging the gap between image-based feature extraction successes and the needs of 3D LiDAR data processing.

Methods

LodoNet's core innovation lies in its strategy to transform the 3D LiDAR problem into a 2D image-based one. The process involves several key steps:

  1. Spherical Projection and Depth Completion: Raw 3D LiDAR point clouds, which are inherently sparse, are projected onto a spherical image. This projection helps to create a more dense representation. To further address sparsity, a depth completion technique is employed, filling in missing depth values by interpolating from nearest valid pixels. Histogram equalization is then applied to enhance the contrast of these spherical images.
  2. 2D Keypoint Detection and Matching: Inspired by image processing, the Scale-Invariant Feature Transform (SIFT) algorithm is used to detect and extract keypoints from the processed spherical images. These keypoints, along with their descriptors, are then matched between consecutive LiDAR frames to form Matched Keypoint Pairs (MKPs).
  3. Projection Back to 3D: The detected 2D MKPs are projected back into the 3D LiDAR space using an inverse projection function. This step ensures that the odometry estimation is performed in the correct 3D coordinate system.
  4. MKPs Selection Module: Not all MKPs are suitable for odometry estimation, especially those from dynamic objects. A PointNet-based segmentation module is introduced to filter out these inappropriate MKPs, selecting only the most reliable ones for the subsequent estimation steps.
  5. Rotation and Translation Estimation Modules: The selected MKPs are fed into two separate deep neural network modules:
  6. Rotation Estimation Module: This module, inspired by PointNet classification, is adapted for regression to estimate the 3-DoF relative rotation. It incorporates a data augmentation strategy for rotation to handle scenarios where straight-line motion dominates training data.
  7. Translation Estimation Module: This module directly estimates the 3-DoF translation using the selected MKPs.

The entire pipeline is designed to decouple the keypoint matching from the odometry regression, aiming for improved convergence and efficiency.

Datasets

The primary dataset used for training and evaluation is the KITTI Odometry dataset. The authors utilize sequences 0-6, 9, and 10 for training, and sequences 7 and 8 for validation. The paper notes that the KITTI dataset provides ground truth odometry poses, which are crucial for training and evaluating the proposed method.

Key Insights and Ideas

  • Leveraging Image Feature Extractors for LiDAR: The most significant insight is the successful transfer of image-based feature extraction techniques (like SIFT) to LiDAR data by converting it into an image-like representation. This bypasses the computational burden of direct 3D point cloud processing.
  • Decoupling Keypoint Matching and Odometry: Explicitly separating the MKP extraction (using SIFT) from the odometry estimation (using deep networks) simplifies the learning problem and improves convergence, avoiding the iterative coupling seen in some ICP-based learning methods.
  • Importance of MKP Filtering: The realization that not all detected keypoints are useful for odometry, and the development of a PointNet-based module to filter dynamic object points, is crucial for improving robustness and accuracy.
  • Spherical Projection for Density: The use of spherical projection is a clever way to create a more uniform and dense representation of sparse LiDAR data, making it more amenable to 2D feature extraction.

Key Results

LodoNet achieves competitive results on the KITTI odometry benchmark. The paper reports that LodoNet performs "on par with or even better results than the state-of-the-art." Specifically, Table 1 shows that LodoNet achieves low translational RMSE (t_rel) and rotational RMSE (r_rel) across various sequences, often outperforming other deep learning methods and even some traditional approaches like LOAM in specific metrics. For instance, on sequence 07, LodoNet achieves a t_rel of 1.70 and r_rel of 0.89, which are highly competitive. The paper also highlights that the proposed method can be easily deployed due to its reliance on a conventional feature descriptor and a lightweight neural network.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while LodoNet performs well, the state-of-the-art geometry-based approach, LOAM, still remains the best option in some aspects.
  • The reliance on SIFT, a traditional feature descriptor, means that the performance is inherently tied to its capabilities and limitations.
  • The "fake" MKPs generated by depth completion need to be carefully handled and removed.
  • Future Work:
    • The authors suggest exploring ways to further integrate the MKP extraction and odometry estimation steps. This could involve end-to-end learning of both components for potentially even greater accuracy and efficiency.
  • Investigating more advanced feature descriptors or learning-based feature extraction directly from the spherical images could be a promising direction.

Summary updated

Abstract

Deep learning based LiDAR odometry (LO) estimation attracts increasing research interests in the field of autonomous driving and robotics. Existing works feed consecutive LiDAR frames into neural networks as point clouds and match pairs in the learned feature space. In contrast, motivated by the success of image based feature extractors, we propose to transfer the LiDAR frames to image space and reformulate the problem as image feature extraction. With the help of scale-invariant feature transform (SIFT) for feature extraction, we are able to generate matched keypoint pairs (MKPs) that can be precisely returned to the 3D space. A convolutional neural network pipeline is designed for LiDAR odometry estimation by extracted MKPs. The proposed scheme, namely LodoNet, is then evaluated in the KITTI odometry estimation benchmark, achieving on par with or even better results than the state-of-the-art.
BibTeX
@inproceedings{zheng2020lodonet,
  author = {Ce Zheng and Yecheng Lyu and Ming Li and Ziming Zhang},
  title = {{LodoNet: A Deep Neural Network with 2D Keypoint Matchingfor 3D LiDAR Odometry Estimation}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=IZ5twgWqWL},
}
Figure