ASSIST-3D: Adapted Scene Synthesis for Class-Agnostic 3D Instance Segmentation

Shengchao Zhou, Jiehong Lin, Jiahui Liu, Shizhen Zhao, Chirui Chang, Xiaojuan Qi

Abstract

Class-agnostic 3D instance segmentation aims to identify all individual objects in a 3D scan, regardless of their semantic category. However, the scarcity of diverse, accurately annotated 3D data hinders the development of robust models. We present ASSIST-3D, a novel synthetic data generation pipeline that leverages massive CAD libraries, LLM-guided spatial reasoning, and realistic sensor simulation. By generating diverse training data that breaks conventional class co-occurrence patterns, ASSIST-3D enables models to learn fundamental object geometry. Our approach significantly improves performance on real-world benchmarks, nearly doubling Average Precision on ScanNet++ compared to models trained only on real data.

Introduction

Instance segmentation is a cornerstone of 3D computer vision, yet most existing methods are "class-aware," limited to a small set of predefined categories. In contrast, "class-agnostic" segmentation seeks to delineate every object in a scene. The primary challenge is the lack of high-quality training data. Real-world datasets are difficult to annotate at scale, while existing synthetic datasets often lack geometric diversity, context complexity, or physical plausibility.

Overview of ASSIST-3D compared to traditional training paradigms.

ASSIST-3D addresses these limitations by "adapting" scene synthesis to teach models the core concept of an "object" through three key innovations: heterogeneous object selection, LLM-driven layout generation, and realistic point cloud construction.

Methodology

The ASSIST-3D pipeline is designed to bridge the gap between synthetic meshes and real-world sensor data while maximizing geometric variety.

The ASSIST-3D pipeline: from object selection to point cloud construction.

1. Heterogeneous Object Selection

To prevent models from relying on predictable object pairings (e.g., a monitor always next to a keyboard), ASSIST-3D utilizes a randomized sampling strategy from the Objaverse library, which contains over 50,000 CAD models. This forces the model to learn object boundaries based on intrinsic geometry rather than semantic context.

2. Scene Layout Generation (LLM + DFS)

Ensuring physical plausibility in synthetic scenes is critical. ASSIST-3D employs GPT-4 to infer logical spatial relationships (e.g., "books go on shelves"). Since LLMs lack precise coordinate reasoning, their logical output is grounded using a Depth-First Search (DFS) algorithm that places objects in a grid-based layout while preventing collisions.

3. Realistic Point Cloud Construction

Synthetic data is often "too perfect" compared to noisy real-world scans. ASSIST-3D simulates the scanning process by rendering 60 virtual RGB-D images from various vantage points within the synthetic scene. These images are fused into a "messy" point cloud that replicates the noise, occlusions, and varying densities found in actual LiDAR or RGB-D captures.

Experimental Results

We evaluated ASSIST-3D by augmenting the training of a Mask3D model and testing it on major indoor benchmarks: ScanNetV2, ScanNet++, and S3DIS.

  • ScanNet++ Performance: The model achieved an Average Precision (AP) of 22.2, a significant jump from the 12.0 AP baseline trained solely on real data.
  • S3DIS Generalization: Performance increased from 13.6 to 29.0 AP, demonstrating the synthetic data's ability to enhance generalization across different environments.
  • Comparison with State-of-the-Art: ASSIST-3D consistently outperformed existing methods like SAI3D and SA3DIP.

Ablation studies confirmed that all three components—diversity, layout logic, and realistic rendering—are essential. Removing any single pillar resulted in a measurable performance decline.

Discussion and Conclusion

ASSIST-3D demonstrates that "diversity over realism" in object placement can actually improve model robustness by preventing over-reliance on semantic priors. Furthermore, it highlights the potential of LLMs as "spatial architects" when paired with traditional search algorithms.

While effective, the pipeline's reliance on GPT-4 introduces computational costs, and the current focus remains on indoor scenes. Future work will explore scaling the synthesis to outdoor environments and further narrowing the domain gap between synthetic noise and real-world sensor artifacts. ASSIST-3D provides a scalable path forward for class-agnostic 3D perception, moving beyond the constraints of manually annotated datasets.

Summary updated

BibTeX
@inproceedings{zhou2026assist,
  author = {Shengchao Zhou and Jiehong Lin and Jiahui Liu and Shizhen Zhao and Chirui Chang and Xiaojuan Qi},
  title = {{ASSIST-3D: Adapted Scene Synthesis for Class-Agnostic 3D Instance Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i16.38384},
}
Figure