PanopticRecon: Leverage Open-vocabulary Instance Segmentation for Zero-shot Panoptic Reconstruction

Xuan Yu, Yili Liu, Chenrui Han, Sitong Mao, Shunbo Zhou, Rong Xiong, Yiyi Liao, Yue Wang

Introduction

Panoptic reconstruction represents a significant frontier in computer vision, aiming to simultaneously recover a scene's 3D geometry while identifying semantic classes and individual object instances. Traditionally, this task has relied on "closed-vocabulary" models—systems trained on a fixed set of categories (e.g., the 20 classes in ScanNet). While effective in controlled environments, these models are inherently brittle when deployed in the real world, where they encounter objects and categories they were never trained to recognize.

PanopticRecon introduces a novel framework that shifts this paradigm toward "open-vocabulary" reconstruction. By leveraging the power of 2D Vision-Language Models (VLMs) and self-supervised feature distillation, PanopticRecon achieves high-fidelity 3D panoptic reconstruction without requiring human-annotated 3D training data. This zero-shot capability allows the system to identify and reconstruct any object described by a text prompt, overcoming the limitations of static datasets.

Overcoming the Hurdles of Zero-Shot 3D Vision

While 2D VLMs like Grounded-SAM enable open-vocabulary recognition, translating their 2D predictions into a consistent 3D space introduces two primary challenges:

  1. Partial Labeling: VLMs typically only label pixels that explicitly match a text prompt. This leaves a significant portion of the scene "unknown," making it difficult to supervise a continuous 3D semantic field.
  2. Instance Association: In a video sequence, a single object (e.g., a chair) will receive different instance IDs across different frames (e.g., Instance #5 in Frame 1 vs. Instance #12 in Frame 10). Without a robust mechanism to link these IDs across time and space, the resulting 3D model will be fragmented.

The PanopticRecon Pipeline

PanopticRecon addresses these issues through a two-stage pipeline that converts noisy, partial 2D labels into a clean, globally consistent 3D panoptic mesh.

Neural Implicit Representation

The system utilizes a hierarchical hash encoding (similar to Instant-NGP) to represent the scene as a Signed Distance Function (SDF). This implicit representation allows for the simultaneous optimization of high-quality geometry alongside semantic and instance fields.

Label Propagation via DINOv2

To solve the partial labeling problem, the framework distills features from DINOv2, a powerful self-supervised model, into the 3D representation. Because DINOv2 features naturally cluster based on semantic and textural similarity, the system can "propagate" labels from a few prompted pixels to all other pixels sharing similar DINOv2 signatures. This ensures that even unprompted regions receive accurate semantic assignments based on their visual characteristics.

3D Instance Graph Segmentation

Rather than relying on fragile frame-to-frame tracking, PanopticRecon builds a global 3D instance graph. The initial 3D mesh is clustered into "superfaces" (nodes). The system then uses 2D masks from Grounded-SAM to "vote" on which nodes belong to the same instance. By formulating instance association as a graph segmentation problem, the model achieves global ID consistency, even when objects are occluded or viewed from disparate angles.

Experimental Results

The authors evaluated PanopticRecon on two rigorous benchmarks: ScanNet V2 (indoor) and KITTI-360 (outdoor driving). In both scenarios, PanopticRecon significantly outperformed existing open-vocabulary state-of-the-art (SOTA) methods.

  • ScanNet V2: PanopticRecon achieved a Panoptic Quality (PQ) of 62.39, more than doubling the performance of the previous SOTA, PVLFF (25.14). Remarkably, it even outperformed some closed-vocabulary methods that had the advantage of training on the specific dataset.
  • KITTI-360: In the challenging outdoor environment characterized by sparse viewpoints and high instance counts, the model achieved a PQ of 35.95, nearly doubling the performance of "Panoptic Lifting" (20.86).

Ablation studies confirmed that the integration of DINOv2 improved the mean Intersection over Union (mIoU) by approximately 3-4%, while the graph-based association proved vastly superior to standard linear assignment or tracking-based approaches.

Key Insights and Future Directions

PanopticRecon demonstrates that geometry can serve as a powerful prior for instance association. By projecting 3D superfaces back into 2D frames, the model creates a robust bridge between 2D masks and 3D objects. Furthermore, the use of DINOv2 as a semantic bridge allows the model to generalize beyond the specific text prompts provided by a user.

Despite its success, the method currently requires human-provided text prompts for classes of interest. Future work may focus on a fully autonomous "discovery" mode for unknown objects. Additionally, while the graph inference is efficient, the two-stage reconstruction process is not yet real-time, presenting an opportunity for optimization in high-speed robotics applications.

In conclusion, PanopticRecon sets a new benchmark for zero-shot 3D scene understanding, proving that open-vocabulary models can achieve levels of consistency and accuracy previously reserved for supervised, closed-vocabulary systems.

Summary updated

Abstract

Panoptic reconstruction is a challenging task in 3D scene understanding. However, most existing methods heavily rely on pre-trained semantic segmentation models and known 3D object bounding boxes for 3D panoptic segmentation, which is not available for in-the-wild scenes. In this paper, we propose a novel zero-shot panoptic reconstruction method from RGB-D images of scenes. For zero-shot segmentation, we leverage open-vocabulary instance segmentation, but it has to face partial labeling and instance association challenges. We tackle both challenges by propagating partial labels with the aid of dense generalized features and building a 3D instance graph for associating 2D instance IDs. Specifically, we exploit partial labels to learn a classifier for generalized semantic features to provide complete labels for scenes with dense distilled features. Moreover, we formulate instance association as a 3D instance graph segmentation problem, allowing us to fully utilize the scene geometry prior and all 2D instance masks to infer global unique pseudo 3D instance ID. Our method outperforms state-of-the-art methods on the indoor dataset ScanNet V2 and the outdoor dataset KITTI-360, demonstrating the effectiveness of our graph segmentation method and reconstruction network.
BibTeX
@inproceedings{yu2024panopticrecon,
  author = {Xuan Yu and Yili Liu and Chenrui Han and Sitong Mao and Shunbo Zhou and Rong Xiong and Yiyi Liao and Yue Wang},
  title = {{PanopticRecon: Leverage Open-vocabulary Instance Segmentation for Zero-shot Panoptic Reconstruction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=iiMtK8JBk8},
}
Figure