Details Matter for Indoor Open-vocabulary 3D Instance Segmentation

Sanghun Jung, Jingjing Zheng, Ke Zhang, Nan Qiao, Albert Y. C. Chen, Lu Xia, Chi Liu, Yuyin Sun, Xiao Zeng, Hsiang-Wei Huang, Byron Boots, Min Sun, Cheng-Hao Kuo

Introduction

Open-vocabulary 3D Instance Segmentation (OV-3DIS) is a critical task for robotics and spatial computing, enabling systems to identify and segment individual objects in a 3D environment based on arbitrary natural language descriptions. Unlike traditional closed-vocabulary methods, OV-3DIS allows a user to query a scene for functional categories (e.g., "something to drink water from") or specific attributes (e.g., "the red chair") without pre-defined labels.

Qualitative examples of open-vocabulary queries like 'Drink Water' and 'Heat Mac & Cheese'.

Despite recent progress, existing pipelines often struggle with "detail leakage," where noisy proposals, context contamination in vision-language models, and uncalibrated similarity scores lead to poor segmentation performance. This paper introduces a refined "recipe" that addresses these issues through hybrid proposal generation, robust tracking, and improved classification techniques.

The Problem: Detail Leakage in 3D Pipelines

Current OV-3DIS methods typically follow a two-stage process: generating 3D object proposals and subsequently labeling them using a Vision-Language Model (VLM) like CLIP. However, several factors degrade performance:

  1. Noisy Proposals: Lifting 2D masks to 3D often results in overlapping, partial, or redundant masks.
  2. Context Contamination: Standard CLIP processing often includes background clutter in image crops, confusing the model's focus on the target object.
  3. Uncalibrated Scores: CLIP similarity scores are not naturally comparable across different queries, making it difficult to establish a universal confidence threshold for filtering false positives.

Methodology

The proposed framework argues that "details matter" and introduces several key innovations to refine the 3D instance segmentation pipeline.

Hybrid Proposal Generation

To ensure high recall for both common and rare objects, the method combines 3D proposals from two sources: 2D image-based masks (via Grounded SAM) and raw 3D point cloud proposals (via class-agnostic networks). This hybrid approach leverages the semantic richness of 2D models and the geometric precision of 3D-native methods.

Robust Tracking and Iterative Merging

To convert 2D masks into coherent 3D instances, the system employs a superpoint-level sIOU (intersection-over-union) metric for frame-to-frame tracking. This is followed by an Iterative Merging and Removal step, which fuses partial masks and deletes redundant inclusions to produce a clean set of final proposals.

Overview of the 3D tracking and iterative merging pipeline.

Refined Classification with Alpha-CLIP and SMS

To improve classification accuracy, the authors replace standard CLIP with Alpha-CLIP, which utilizes an object mask as an alpha channel. This forces the model to ignore background clutter and focus strictly on the target points. Additionally, the authors introduce Standardized Maximum Similarity (SMS). By calculating the mean and variance of similarity scores for a specific query within a scene, they convert raw scores into standardized Z-scores. This allows for a consistent filtering threshold across diverse text queries, significantly reducing false positives.

Results and Key Insights

The framework achieves new state-of-the-art (SoTA) performance across major benchmarks:

  • ScanNet200: Reached 25.8 mAP, nearly doubling the performance of previous leaders like OpenMask3D (15.4 mAP).
  • S3DIS: Outperformed the previous best (Open3DIS) by approximately 9% in mAP.
  • Generalization: The model showed remarkable success on synthetic datasets (Replica) and fine-grained datasets (ScanNet++), even surpassing some closed-vocabulary models trained on specific labels.

The authors found that image-based proposals excel at identifying functional objects, while point-cloud proposals are superior for geometric accuracy. The combination of these strengths, paired with the standardized scoring of SMS, allows the model to handle complex, real-world indoor environments with unprecedented precision.

Conclusion

By focusing on the technical details of proposal merging and score calibration, this work demonstrates that existing foundation models can be significantly more effective for 3D tasks. While the pipeline is computationally intensive due to its reliance on heavy 2D models, it sets a new standard for open-vocabulary understanding of 3D spaces.

Summary updated

BibTeX
@inproceedings{jung2025details,
  author = {Sanghun Jung and Jingjing Zheng and Ke Zhang and Nan Qiao and Albert Y. C. Chen and Lu Xia and Chi Liu and Yuyin Sun and Xiao Zeng and Hsiang-Wei Huang and Byron Boots and Min Sun and Cheng-Hao Kuo},
  title = {{Details Matter for Indoor Open-vocabulary 3D Instance Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure