GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Network

Ryan Razani, Ran Cheng, Enxu Li, Ehsan Taghavi, Yuan Ren, Bingbing Liu

Introduction

Panoptic segmentation represents a significant challenge in 3D scene understanding, requiring the simultaneous classification of "stuff" (uncountable background elements like roads and vegetation) and "things" (countable objects like cars and pedestrians). In the context of LiDAR data, existing methods typically follow either proposal-based or proposal-free paradigms. Proposal-based methods rely on complex detection networks to find bounding boxes, while proposal-free methods often struggle with the clustering phase.

GP-S3Net addresses a fundamental flaw in proposal-free methods: the reliance on unsupervised clustering (e.g., HDBSCAN). Traditional clustering requires manual tuning of geometric parameters, which frequently leads to over-segmentation (splitting a single object) or under-segmentation (merging distinct nearby objects). GP-S3Net reimagines this as a supervised edge classification problem within a graph-based framework.

Methodology

GP-S3Net employs a bottom-up pipeline that transforms point-wise semantic labels into coherent panoptic instances through a supervised graph convolutional network (GCNN).

The GP-S3Net workflow: from point cloud over-segmentation to supervised graph-based instance merging.

The Pipeline Architecture

  1. Semantic Backbone: The system utilizes (AF)$^2$-S3Net, a 3D sparse convolutional network, to generate high-quality semantic labels for the entire point cloud.
  2. Over-Segmentation: Points identified as "things" are initially grouped using HDBSCAN. The objective is to produce fine-grained clusters where no single cluster contains points from more than one object.
  3. Cluster Embedding: Each cluster is processed via 3D sparse convolution and global average pooling to generate a unique feature vector (embedding).
  4. EdgeNet (GCNN): Clusters are treated as nodes in a fully connected graph. The EdgeNet model, utilizing SageConv layers, is trained to predict the connectivity between these nodes. Unlike traditional heuristics, EdgeNet learns the specific features that indicate whether two clusters belong to the same physical object.
  5. Panoptic Fusion: Edges predicted as "connected" are merged, and the resulting instances are combined with the "stuff" predictions from the backbone. A majority voting scheme ensures consistency between the final semantic and instance labels.

Experimental Results

The researchers evaluated GP-S3Net on major benchmarks including SemanticKITTI, nuScenes, and SemanticPOSS. At the time of publication, the model achieved the #1 position on the SemanticKITTI leaderboard.

Qualitative comparison showing GP-S3Net's superior instance segmentation compared to DS-Net and TORNADO-Net.

Key Performance Metrics

  • State-of-the-Art Accuracy: GP-S3Net achieved a 60.0% Panoptic Quality (PQ) score on SemanticKITTI, a +4.1% improvement over the previous state-of-the-art (DS-Net).
  • Instance Robustness: The model demonstrated a unique ability to maintain separation between pedestrians in close proximity and to correctly merge large, fragmented objects like buses and trucks—common failure points for offset-based methods.
  • Semantic Integrity: The panoptic branch did not compromise semantic accuracy, maintaining a 70.8% mIoU on the SemanticKITTI test set.

Conclusion

GP-S3Net demonstrates that shifting from unsupervised geometric heuristics to supervised graph-based clustering significantly improves LiDAR panoptic segmentation. By treating over-segmented clusters as graph nodes, the model achieves high efficiency suitable for real-time robotics while providing the robustness necessary for complex, crowded environments. Future work may involve reducing the quadratic complexity of the fully connected graph and exploring the application of this supervised clustering approach to other modalities like RGB imagery.

Summary updated

Abstract

Panoptic segmentation as an integrated task of both static environmental understanding and dynamic object identification, has recently begun to receive broad research interest. In this paper, we propose a new computationally efficient LiDAR based panoptic segmentation framework, called GP-S3Net. GP-S3Net is a proposal-free approach in which no object proposals are needed to identify the objects in contrast to conventional two-stage panoptic systems, where a detection network is incorporated for capturing instance information. Our new design consists of a novel instance-level network to process the semantic results by constructing a graph convolutional network to identify objects (foreground), which later on are fused with the background classes. Through the fine-grained clusters of the foreground objects from the semantic segmentation backbone, over-segmentation priors are generated and subsequently processed by 3D sparse convolution to embed each cluster. Each cluster is treated as a node in the graph and its corresponding embedding is used as its node feature. Then a GCNN predicts whether edges exist between each cluster pair. We utilize the instance label to generate ground truth edge labels for each constructed graph in order to supervise the learning. Extensive experiments demonstrate that GP-S3Net outperforms the current state-of-the-art approaches, by a significant margin across available datasets such as, nuScenes and SemanticPOSS, ranking first on the competitive public SemanticKITTI leaderboard upon publication.
BibTeX
@inproceedings{razani2021gp,
  author = {Ryan Razani and Ran Cheng and Enxu Li and Ehsan Taghavi and Yuan Ren and Bingbing Liu},
  title = {{GP-S3Net: Graph-based Panoptic Sparse Semantic Segmentation Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
}
Figure