TL;DR
This paper introduces DQFormer, a novel unified framework for LiDAR panoptic segmentation that decouples "thing" and "stuff" queries. By treating these distinct object types separately, DQFormer effectively mitigates mutual competition and ambiguity, leading to state-of-the-art performance on challenging LiDAR datasets.
What Problem Does This Paper Address?
LiDAR panoptic segmentation is crucial for autonomous driving and robotics, aiming to simultaneously identify and delineate both individual objects ("things" like cars and pedestrians) and background regions ("stuff" like roads and vegetation) in 3D point clouds. Existing methods often struggle with this task due to two main challenges:
- Mutual Competition: "Things" and "stuff" have vastly different spatial distributions and characteristics. "Stuff" is typically widespread, while "things" are localized. This difference can cause algorithms designed for one to interfere with the segmentation of the other, especially when using unified query-based approaches.
- Classification/Segmentation Ambiguity: In unified models, the process of assigning a semantic label and segmenting an instance can become ambiguous, leading to errors in both tasks.
The paper aims to address these challenges by proposing a unified framework that explicitly accounts for the distinct properties of "things" and "stuff."
Methods
DQFormer is a unified query-based framework that tackles LiDAR panoptic segmentation with a novel approach:
- Decoupled Queries: The core innovation is the decoupling of "thing" and "stuff" queries. Instead of a single set of queries, DQFormer generates separate query sets tailored to the characteristics of each class type.
- Multi-scale Feature Encoder: A backbone network extracts voxel-level features and point-wise embeddings at multiple resolutions from the input LiDAR point cloud.
- Decoupled Query Generator: This module is responsible for creating informative queries.
- Things Query Generation: It leverages Bird's Eye View (BEV) embeddings to precisely locate potential "thing" instances based on predicted object centers.
- Stuff Query Generation: It uses learnable queries that attend to BEV features to capture the broader spatial context of "stuff" regions.
- Query Fusion: Queries from different resolutions are fused to create richer representations for both "things" and "stuff." This fusion is done separately for each class type, ensuring semantic consistency.
- Query-Oriented Mask Decoder: A transformer-based decoder uses the decoupled and fused queries to perform masked cross-attention with point embeddings. This allows the queries to guide the segmentation process, generating precise masks for both "things" and "stuff" within a unified workflow.
Datasets
The proposed DQFormer framework is evaluated on two widely used LiDAR datasets:
- nuScenes: A comprehensive urban driving dataset with 1000 LiDAR scenes, featuring 16 annotated point-wise labels (10 thing categories, 6 stuff categories).
- SemanticKITTI: A dataset derived from the KITTI odometry dataset, containing 22 LiDAR sequences with 19 annotated point-wise labels (8 thing classes, 11 stuff classes).
Key Insights and Ideas
- Decoupling is Key: The distinct nature of "things" and "stuff" in LiDAR scenes is not just a detail but a fundamental property that requires separate handling for optimal segmentation. Treating them as one can lead to performance degradation.
- BEV for Localization: Bird's Eye View (BEV) representations are highly effective for localizing "thing" instances due to their concentrated nature.
- Global Context for Stuff: "Stuff" segmentation benefits from capturing broader scene context, which is achieved through dedicated "stuff" queries.
- Unified Workflow Efficiency: By integrating semantic and instance segmentation into a single query-based pipeline, DQFormer achieves high performance without the computational overhead of separate branches or multi-stage approaches.
- Mask Fusion Importance: For large objects, merging duplicated instance masks (post-fusion) is crucial to avoid over-segmentation and ensure a clean final output.
Key Results
DQFormer demonstrates state-of-the-art performance on both nuScenes and SemanticKITTI datasets:
- nuScenes: Achieved superior Panoptic Quality (PQ) compared to existing methods, outperforming detection-based methods like EfficientLPS by a significant margin (+17.2% in PQTh) and query-based methods like MaskPLS by 12.8% on the test split.
- SemanticKITTI: Showed substantial improvements, outperforming detection-based and clustering-based methods, and achieving 1.5% higher PQ than CenterLPS. It also surpassed query-based methods like MaskPLS and PUPS.
- Efficiency: DQFormer achieves competitive or faster inference speeds compared to existing methods, particularly outperforming detection-based and clustering-based approaches.
Limitations and Future Work
- Failure Case in Query Fusion: The paper notes a failure case where queries related to a large truck were not fully fused by the query fusion module, highlighting the importance of the subsequent mask fusion step.
- Future Work: While not explicitly detailed, the success of DQFormer suggests that further exploration of query generation strategies, attention mechanisms, and fusion techniques tailored to specific LiDAR characteristics could lead to even greater improvements. The paper's focus on decoupling also opens avenues for investigating how this principle can be applied to other multi-task LiDAR perception problems.
Summary updated