Real-time semantic segmentation of LiDAR data is a "holy grail" for autonomous robotics. While we have powerful models that can identify every car, pedestrian, and sidewalk in a 3D point cloud, they often require massive GPUs that simply won't fit (or won't run efficiently) on an embedded platform like an NVIDIA Jetson.
In their paper, "Multi-scale Interaction for Real-time LiDAR Data Segmentation on an Embedded Platform," Shijie Li and his team introduce MINet (Multi-scale Interaction Network). This lightweight model manages to outperform state-of-the-art methods while using 98% fewer parameters, making it a perfect candidate for real-world deployment.
TL;DR
MINet is a highly efficient, projection-based neural network designed for real-time LiDAR segmentation on embedded hardware. It achieves competitive accuracy with just 1 million parameters—a 50x reduction compared to previous leaders—and runs at over 24 FPS on a Jetson AGX, comfortably exceeding standard sensor frame rates.
What Problem Does This Paper Address?
LiDAR sensors generate massive, unordered point clouds. Processing these directly (point-based methods) involves complex spatial operations that are computationally expensive and hard to optimize for embedded chips.
An alternative is "projection-based" methods, which project the 3D points onto a 2D spherical map, allowing the use of standard, fast 2D convolutions. However, existing projection models face a dilemma: they are either too small to be accurate or so large (50M+ parameters) that they crawl on embedded hardware. The challenge is finding the "Goldilocks" zone—a model that is lightweight enough for a robot but powerful enough to navigate a busy street safely.
Methods
The authors built MINet around three core architectural innovations:
- Multi-scale Interaction Module (MIM): Instead of one giant processing path, MINet uses three parallel paths at different resolutions. Crucially, it balances resources: the high-resolution path uses lightweight "MobileBlocks," while the low-resolution path uses more complex "BasicBlocks" to extract deep semantic meaning.
- Dense Top-to-Bottom Interaction: To avoid redundant calculations, feature maps from the high-resolution paths are passed down to all lower paths. This ensures that the deeper layers benefit from the fine-grained details captured early on.
- Mini Fusion Module (MFM): LiDAR data isn't just one thing; it's a mix of 3D coordinates (x, y, z), remission (intensity), and depth. MINet processes these modalities independently at first to "calibrate" them before fusing them into a single feature space.
To train the model, they used a "Booster Training Strategy." This involves adding intermediate semantic supervision to the top paths and a specialized edge supervision loss to the final fusion stage. This edge loss is critical for sharpening object boundaries that often get "fuzzy" during the upsampling process.
Datasets
The researchers evaluated MINet on two major benchmarks:
- SemanticKITTI: The industry standard, featuring over 43,000 scans from outdoor driving environments.
- SemanticPOSS: A smaller but more challenging dataset with sparser point clouds, providing a good test for model robustness.
Key Insights and Ideas
- Asymmetric Complexity: One of the biggest "aha" moments is that you shouldn't treat every scale the same. By using simpler operations on high-res data and complex operations on low-res data, you get the best of both worlds: speed and semantic depth.
- The Power of Edge Loss: Standard cross-entropy loss often struggles with object boundaries. By explicitly supervising the "edges" of objects during training, the model produces much cleaner segmentations without adding any computational cost during inference.
- Modality Independence: Treating different LiDAR data types (like intensity vs. distance) as separate entities initially, rather than just stacking them as channels, significantly improves the final fusion quality.
Key Results
MINet sets a new bar for efficiency-to-accuracy trade-offs:
- Extreme Efficiency: It uses only 1 million parameters. For context, the popular RangeNet53 uses over 50 million.
- Embedded Performance: On an NVIDIA Jetson AGX, MINet processes 24-80 scans per second (depending on resolution). This is 4x faster than RangeNet53.
- High Accuracy: On SemanticKITTI, it achieved a mean Intersection-over-Union (mIoU) of 52.4% (rising to 55.2% with k-NN post-processing), outperforming many models that are significantly larger and slower.
Limitations and Future Work
While MINet is a leap forward for real-time deployment, the authors note a few areas for growth:
- Temporal Information: The current model processes single scans. Integrating information from multiple consecutive frames (temporal fusion) could further improve accuracy, especially for moving objects.
- Task Generalization: The principles used here—multi-scale interaction and balanced computation—could likely be applied to other 3D tasks like object detection or motion prediction.
For those interested in the implementation, the authors have made their source code available on GitHub.
Summary updated