Multi-Scale Interaction for Real-Time LiDAR Data Segmentation on an Embedded Platform

Shijie Li, Xieyuanli Chen, Yun Liu, Dengxin Dai, Cyrill Stachniss, Juergen Gall

Real-time semantic segmentation of LiDAR data is a "holy grail" for autonomous robotics. While we have powerful models that can identify every car, pedestrian, and sidewalk in a 3D point cloud, they often require massive GPUs that simply won't fit (or won't run efficiently) on an embedded platform like an NVIDIA Jetson.

In their paper, "Multi-scale Interaction for Real-time LiDAR Data Segmentation on an Embedded Platform," Shijie Li and his team introduce MINet (Multi-scale Interaction Network). This lightweight model manages to outperform state-of-the-art methods while using 98% fewer parameters, making it a perfect candidate for real-world deployment.

TL;DR

MINet is a highly efficient, projection-based neural network designed for real-time LiDAR segmentation on embedded hardware. It achieves competitive accuracy with just 1 million parameters—a 50x reduction compared to previous leaders—and runs at over 24 FPS on a Jetson AGX, comfortably exceeding standard sensor frame rates.

What Problem Does This Paper Address?

LiDAR sensors generate massive, unordered point clouds. Processing these directly (point-based methods) involves complex spatial operations that are computationally expensive and hard to optimize for embedded chips.

An alternative is "projection-based" methods, which project the 3D points onto a 2D spherical map, allowing the use of standard, fast 2D convolutions. However, existing projection models face a dilemma: they are either too small to be accurate or so large (50M+ parameters) that they crawl on embedded hardware. The challenge is finding the "Goldilocks" zone—a model that is lightweight enough for a robot but powerful enough to navigate a busy street safely.

Methods

The authors built MINet around three core architectural innovations:

  1. Multi-scale Interaction Module (MIM): Instead of one giant processing path, MINet uses three parallel paths at different resolutions. Crucially, it balances resources: the high-resolution path uses lightweight "MobileBlocks," while the low-resolution path uses more complex "BasicBlocks" to extract deep semantic meaning.
  2. Dense Top-to-Bottom Interaction: To avoid redundant calculations, feature maps from the high-resolution paths are passed down to all lower paths. This ensures that the deeper layers benefit from the fine-grained details captured early on.
  3. Mini Fusion Module (MFM): LiDAR data isn't just one thing; it's a mix of 3D coordinates (x, y, z), remission (intensity), and depth. MINet processes these modalities independently at first to "calibrate" them before fusing them into a single feature space.
The MINet architecture featuring the Multi-scale Interaction Module and Fusion modules.

To train the model, they used a "Booster Training Strategy." This involves adding intermediate semantic supervision to the top paths and a specialized edge supervision loss to the final fusion stage. This edge loss is critical for sharpening object boundaries that often get "fuzzy" during the upsampling process.

Datasets

The researchers evaluated MINet on two major benchmarks:

  • SemanticKITTI: The industry standard, featuring over 43,000 scans from outdoor driving environments.
  • SemanticPOSS: A smaller but more challenging dataset with sparser point clouds, providing a good test for model robustness.

Key Insights and Ideas

  • Asymmetric Complexity: One of the biggest "aha" moments is that you shouldn't treat every scale the same. By using simpler operations on high-res data and complex operations on low-res data, you get the best of both worlds: speed and semantic depth.
  • The Power of Edge Loss: Standard cross-entropy loss often struggles with object boundaries. By explicitly supervising the "edges" of objects during training, the model produces much cleaner segmentations without adding any computational cost during inference.
  • Modality Independence: Treating different LiDAR data types (like intensity vs. distance) as separate entities initially, rather than just stacking them as channels, significantly improves the final fusion quality.
A comparison of the MobileBlock (top) and BasicBlock (bottom) used in the different paths.

Key Results

MINet sets a new bar for efficiency-to-accuracy trade-offs:

  • Extreme Efficiency: It uses only 1 million parameters. For context, the popular RangeNet53 uses over 50 million.
  • Embedded Performance: On an NVIDIA Jetson AGX, MINet processes 24-80 scans per second (depending on resolution). This is 4x faster than RangeNet53.
  • High Accuracy: On SemanticKITTI, it achieved a mean Intersection-over-Union (mIoU) of 52.4% (rising to 55.2% with k-NN post-processing), outperforming many models that are significantly larger and slower.
Visualization of the trade-off between accuracy (mIoU) and runtime. MINet occupies the top-left "sweet spot."

Limitations and Future Work

While MINet is a leap forward for real-time deployment, the authors note a few areas for growth:

  • Temporal Information: The current model processes single scans. Integrating information from multiple consecutive frames (temporal fusion) could further improve accuracy, especially for moving objects.
  • Task Generalization: The principles used here—multi-scale interaction and balanced computation—could likely be applied to other 3D tasks like object detection or motion prediction.

For those interested in the implementation, the authors have made their source code available on GitHub.

Summary updated

Abstract

Real-time semantic segmentation of LiDAR data is crucial for autonomously driving vehicles and robots, which are usually equipped with an embedded platform and have limited computational resources. Approaches that operate directly on the point cloud use complex spatial aggregation operations, which are very expensive and difficult to deploy on embedded platforms. As an alternative, projection-based methods are more efficient and can run on embedded hardware. However, current projection-based methods either have a low accuracy or require millions of parameters. In this letter, we therefore propose a projection-based method, called Multi-scale Interaction Network (MINet), which is very efficient and accurate. The network uses multiple paths with different scales and balances the computational resources between the scales. Additional dense interactions between the scales avoid redundant computations and make the network highly efficient. The proposed network outperforms point-based, image-based, and projection-based methods in terms of accuracy, number of parameters, and runtime. Moreover, the network processes more than 24 scans, captured by a high-resolution LiDAR sensor with 64 beams, per second on an embedded platform, which is higher than the framerate of the sensor. The network is therefore suitable for robotics applications.
BibTeX
@article{li2022multi,
  author = {Shijie Li and Xieyuanli Chen and Yun Liu and Dengxin Dai and Cyrill Stachniss and Juergen Gall},
  title = {{Multi-Scale Interaction for Real-Time LiDAR Data Segmentation on an Embedded Platform}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=kHCxBlg9nR_},
}
Figure