PANet: LiDAR Panoptic Segmentation with Sparse Instance Proposal and Aggregation

Jianbiao Mei, Yu Yang, Mengmeng Wang, Xiaojun Hou, Laijian Li, Yong Liu

TL;DR

PANet is a novel framework for LiDAR Panoptic Segmentation that replaces the traditional, computationally expensive "offset branch" with a fast, non-learning Sparse Instance Proposal (SIP) module. By combining this with a KNN-Transformer for instance aggregation, PANet effectively resolves over-segmentation of large objects while achieving state-of-the-art results at significantly higher speeds than previous methods.

Introduction: The Challenge of LiDAR Panoptic Segmentation

LiDAR Panoptic Segmentation (LPS) is a critical task for autonomous driving, requiring simultaneous semantic segmentation (identifying object classes) and instance segmentation (distinguishing individual objects).

Most existing approaches utilize a "clustering-based" method where a neural network predicts an "offset" vector from every point to its object center. However, LiDAR data's inherent sparsity and varying density make accurate offset prediction difficult. A common failure mode is "over-segmentation," where large objects like buses or trucks are mistakenly split into multiple smaller fragments. PANet addresses these issues by rethinking the instance grouping process.

Methodology

PANet introduces a streamlined architecture that avoids the unreliable offset head in favor of two primary components: the Sparse Instance Proposal (SIP) module and the Instance Aggregation (IA) module.

The overall framework of PANet and the Sparse Instance Proposal (SIP) module.

Sparse Instance Proposal (SIP) Module

The SIP module is a non-learning component that groups points into instances through a three-step "sampling-shifting-grouping" workflow:

  1. Balanced Point-Sampling (BPS): Space is divided into voxels, and one representative "seed point" is selected per voxel. This ensures a uniform point distribution regardless of distance from the sensor, reducing computational overhead.
  2. Bubble Shrinking (BS): An iterative algorithm that "shrinks" seed points toward cluster centers without requiring a trained network for geometric guidance.
  3. Connected Component Labeling (CCL): A graph-based algorithm that finalizes the grouping of shifted points into initial instance proposals.

Instance Aggregation (IA) Module

To combat the over-segmentation of large objects, the IA module uses a learning-based KNN-Transformer. This module analyzes the proposals from SIP and calculates "affinities" between them. If the model determines that two fragments (e.g., the front and rear of a long truck) belong to the same entity, it merges them into a single instance.

Experimental Results

The researchers evaluated PANet on the SemanticKITTI and nuScenes benchmarks, yielding several key insights:

  • State-of-the-Art Accuracy: PANet achieved a Panoptic Quality (PQ) of 61.7% on the SemanticKITTI validation set, outperforming many complex learning-based clustering methods.
  • Superior Efficiency: The non-learning SIP module is approximately 13x faster than the Dynamic Shifting (DS) modules used in prior state-of-the-art models (19.5ms vs. 259.1ms).
  • Robustness for Large Objects: The IA module significantly improved performance for large vehicle categories. For instance, the PQ for the "Truck" category increased by 3.1% when aggregation was enabled.

Conclusion

By ditching the traditional offset head for a sparse, proposal-based approach, PANet demonstrates that non-learning geometric grouping can be both more efficient and more robust than fully learned alternatives. While challenges remain in extremely sparse long-range data, PANet sets a new standard for real-time, high-accuracy 3D scene understanding.

Summary updated

BibTeX
@inproceedings{mei2023panet,
  author = {Jianbiao Mei and Yu Yang and Mengmeng Wang and Xiaojun Hou and Laijian Li and Yong Liu},
  title = {{PANet: LiDAR Panoptic Segmentation with Sparse Instance Proposal and Aggregation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342468},
}
Figure