Stress-Testing LiDAR Registration

Amnon Drory, Shai Avidan, Raja GiryesarXiv 2022 · DOI · arXiv

Introduction: The Evaluation Gap

Point Cloud Registration (PCR) is a fundamental task in autonomous driving, enabling 6-DOF motion estimation by aligning successive 3D scans. While deep learning has significantly advanced feature matching, the robust estimation stage—responsible for filtering outliers like dynamic objects—remains critical. Current research predominantly relies on the KITTI-10m dataset, which the authors argue is "saturated." In these benchmarks, scans are characterized by high overlap and minimal rotation, leading to near-perfect scores across disparate algorithms and obscuring true performance differences.

Benchmark saturation on KITTI-10m.

Methods: Modernizing a Classic

The study evaluates a standard two-stage pipeline: feature matching using Fully Convolutional Geometric Features (FCGF) followed by robust estimation. The authors compare deep learning-based estimators like DGR and PointDSC against a "modern" RANSAC implementation. This optimized RANSAC incorporates PROSAC for match prioritization, Edge-Length Consistency (ELC) for rapid outlier rejection, and Local Optimization (LO-RANSAC) for refinement.

A key contribution is Grid-Prioritized Filtering (GPF). GPF addresses the issue of match clustering—where an estimator might overfit to a single high-density object—by dividing the 3D space into a grid and ensuring matches are spatially distributed across the entire scene. This prevents local errors from dominating the global alignment.

Challenging New Benchmarks

To move beyond saturated datasets, the authors introduced three "Balanced" registration sets derived from Apollo-Southbay and NuScenes. These datasets use a novel sampling algorithm to ensure a uniform distribution across the 6-DOF motion space, including difficult cases with rotations up to 90 degrees and overlaps as low as 20%. Unlike KITTI-10m, these sets reveal the true limitations of current robust estimators.

Key Results and Insights

The experimental results reveal a "classic comeback." On the challenging NuScenes-Boston set, RANSAC (GPF) combined with ICP achieved a 77.7% recall, significantly outperforming deep learning alternatives like DGR (48.07%) and PointDSC (66.78%).

Recall vs. Speed comparison on challenging benchmarks.

Beyond accuracy, RANSAC variants proved to be the most efficient, with processing times between 0.11 and 0.17 seconds, compared to 0.46 seconds for DGR. The study also highlighted a significant drop in performance (~16% recall) when models trained in one city (Boston) were tested in another (Singapore), suggesting that geometric features may inadvertently memorize location-specific characteristics.

Conclusion

This research underscores the necessity of challenging, non-saturated benchmarks for LiDAR registration. It demonstrates that a well-tuned, spatially-aware RANSAC remains the state-of-the-art for robust estimation in automotive contexts, offering a superior balance of speed and accuracy compared to current deep learning models. Future work should focus on improving the cross-domain generalization of local descriptors to ensure reliability across diverse urban environments.

Summary updated

Abstract

Rigid Point Cloud Registration (PCR) algorithms aim to estimate the 6-DOF relative motion between two point clouds, which is important in various fields, including autonomous driving. Recent years have seen a significant improvement in global PCR algorithms, i.e. algorithms that can handle a large relative motion. This has been demonstrated in various scenarios, including indoor scenes, but has only been minimally tested in the Automotive setting, where point clouds are produced by vehicle-mounted LiDAR sensors. In this work, we aim to answer questions that are important for automotive applications, including: which of the new algorithms is the most accurate, and which is fastest? How transferable are deep-learning approaches, e.g. what happens when you train a network with data from Boston, and run it in a vehicle in Singapore? How small can the overlap between point clouds be before the algorithms start to deteriorate? To what extent are the algorithms rotation invariant? Our results are at times surprising. When comparing robust parameter estimation methods for registration, we find that the fastest and most accurate is not one of the newest approaches. Instead, it is a modern variant of the well known RANSAC technique. We also suggest a new outlier filtering method, Grid-Prioritized Filtering (GPF), to further improve it. An additional contribution of this work is an algorithm for selecting challenging sets of frame-pairs from automotive LiDAR datasets. This enables meaningful benchmarking in the Automotive LiDAR setting, and can also improve training for learning algorithms.
BibTeX
@misc{drory2022stress,
  author = {Amnon Drory and Shai Avidan and Raja Giryes},
  title = {{Stress-Testing LiDAR Registration}},
  year = {2022},
  eprint = {2204.07719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.07719},
}
Figure