DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queries

Yu Yang, Jianbiao Mei, Liang Liu, Siliang Du, Yilin Xiao, Jongwon Ra, Yong Liu, Xiao Xu, Huifeng WuarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces DQFormer, a novel unified framework for LiDAR panoptic segmentation that decouples "thing" and "stuff" queries. By treating these distinct object types separately, DQFormer effectively mitigates mutual competition and ambiguity, leading to state-of-the-art performance on challenging LiDAR datasets.

What Problem Does This Paper Address?

LiDAR panoptic segmentation is crucial for autonomous driving and robotics, aiming to simultaneously identify and delineate both individual objects ("things" like cars and pedestrians) and background regions ("stuff" like roads and vegetation) in 3D point clouds. Existing methods often struggle with this task due to two main challenges:

  1. Mutual Competition: "Things" and "stuff" have vastly different spatial distributions and characteristics. "Stuff" is typically widespread, while "things" are localized. This difference can cause algorithms designed for one to interfere with the segmentation of the other, especially when using unified query-based approaches.
  2. Classification/Segmentation Ambiguity: In unified models, the process of assigning a semantic label and segmenting an instance can become ambiguous, leading to errors in both tasks.

The paper aims to address these challenges by proposing a unified framework that explicitly accounts for the distinct properties of "things" and "stuff."

Methods

DQFormer is a unified query-based framework that tackles LiDAR panoptic segmentation with a novel approach:

  • Decoupled Queries: The core innovation is the decoupling of "thing" and "stuff" queries. Instead of a single set of queries, DQFormer generates separate query sets tailored to the characteristics of each class type.
  • Multi-scale Feature Encoder: A backbone network extracts voxel-level features and point-wise embeddings at multiple resolutions from the input LiDAR point cloud.
  • Decoupled Query Generator: This module is responsible for creating informative queries.
  • Things Query Generation: It leverages Bird's Eye View (BEV) embeddings to precisely locate potential "thing" instances based on predicted object centers.
  • Stuff Query Generation: It uses learnable queries that attend to BEV features to capture the broader spatial context of "stuff" regions.
  • Query Fusion: Queries from different resolutions are fused to create richer representations for both "things" and "stuff." This fusion is done separately for each class type, ensuring semantic consistency.
  • Query-Oriented Mask Decoder: A transformer-based decoder uses the decoupled and fused queries to perform masked cross-attention with point embeddings. This allows the queries to guide the segmentation process, generating precise masks for both "things" and "stuff" within a unified workflow.

Datasets

The proposed DQFormer framework is evaluated on two widely used LiDAR datasets:

  • nuScenes: A comprehensive urban driving dataset with 1000 LiDAR scenes, featuring 16 annotated point-wise labels (10 thing categories, 6 stuff categories).
  • SemanticKITTI: A dataset derived from the KITTI odometry dataset, containing 22 LiDAR sequences with 19 annotated point-wise labels (8 thing classes, 11 stuff classes).

Key Insights and Ideas

  • Decoupling is Key: The distinct nature of "things" and "stuff" in LiDAR scenes is not just a detail but a fundamental property that requires separate handling for optimal segmentation. Treating them as one can lead to performance degradation.
  • BEV for Localization: Bird's Eye View (BEV) representations are highly effective for localizing "thing" instances due to their concentrated nature.
  • Global Context for Stuff: "Stuff" segmentation benefits from capturing broader scene context, which is achieved through dedicated "stuff" queries.
  • Unified Workflow Efficiency: By integrating semantic and instance segmentation into a single query-based pipeline, DQFormer achieves high performance without the computational overhead of separate branches or multi-stage approaches.
  • Mask Fusion Importance: For large objects, merging duplicated instance masks (post-fusion) is crucial to avoid over-segmentation and ensure a clean final output.

Key Results

DQFormer demonstrates state-of-the-art performance on both nuScenes and SemanticKITTI datasets:

  • nuScenes: Achieved superior Panoptic Quality (PQ) compared to existing methods, outperforming detection-based methods like EfficientLPS by a significant margin (+17.2% in PQTh) and query-based methods like MaskPLS by 12.8% on the test split.
  • SemanticKITTI: Showed substantial improvements, outperforming detection-based and clustering-based methods, and achieving 1.5% higher PQ than CenterLPS. It also surpassed query-based methods like MaskPLS and PUPS.
  • Efficiency: DQFormer achieves competitive or faster inference speeds compared to existing methods, particularly outperforming detection-based and clustering-based approaches.

Limitations and Future Work

  • Failure Case in Query Fusion: The paper notes a failure case where queries related to a large truck were not fully fused by the query fusion module, highlighting the importance of the subsequent mask fusion step.
  • Future Work: While not explicitly detailed, the success of DQFormer suggests that further exploration of query generation strategies, attention mechanisms, and fusion techniques tailored to specific LiDAR characteristics could lead to even greater improvements. The paper's focus on decoupling also opens avenues for investigating how this principle can be applied to other multi-task LiDAR perception problems.

Summary updated

Abstract

LiDAR panoptic segmentation, which jointly performs instance and semantic segmentation for things and stuff classes, plays a fundamental role in LiDAR perception tasks. While most existing methods explicitly separate these two segmentation tasks and utilize different branches (i.e., semantic and instance branches), some recent methods have embraced the query-based paradigm to unify LiDAR panoptic segmentation. However, the distinct spatial distribution and inherent characteristics of objects(things) and their surroundings(stuff) in 3D scenes lead to challenges, including the mutual competition of things/stuff and the ambiguity of classification/segmentation. In this paper, we propose decoupling things/stuff queries according to their intrinsic properties for individual decoding and disentangling classification/segmentation to mitigate ambiguity. To this end, we propose a novel framework dubbed DQFormer to implement semantic and instance segmentation in a unified workflow. Specifically, we design a decoupled query generator to propose informative queries with semantics by localizing things/stuff positions and fusing multi-level BEV embeddings. Moreover, a query-oriented mask decoder is introduced to decode corresponding segmentation masks by performing masked cross-attention between queries and mask embeddings. Finally, the decoded masks are combined with the semantics of the queries to produce panoptic results. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the superiority of our DQFormer framework.
BibTeX
@misc{yang2024dqformer,
  author = {Yu Yang and Jianbiao Mei and Liang Liu and Siliang Du and Yilin Xiao and Jongwon Ra and Yong Liu and Xiao Xu and Huifeng Wu},
  title = {{DQFormer: Towards Unified LiDAR Panoptic Segmentation with Decoupled Queries}},
  year = {2024},
  eprint = {2408.15813},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.15813},
}
Figure