Towards Part-Based Understanding of RGB-D Scans

Alexey Bokhovkin, Vladislav Ishimtsev, Emil Bogomolov, Denis Zorin, Alexey Artemov, Evgeny Burnaev, Angela Dai

Introduction

While 3D scene understanding has made significant progress in object-level identification and instance segmentation, most existing methods treat objects as monolithic entities. However, human interaction with the physical world is fundamentally part-based: we sit on a chair's seat, pull a cabinet's handle, or place objects on a table's top. Understanding these functional components is critical for robotics and augmented reality.

Real-world RGB-D scans (e.g., from Kinect or mobile sensors) are often noisy, cluttered, and incomplete due to occlusions. This paper introduces the task of semantic part completion, which aims to predict the full, fine-grained part structure of objects even when significant portions are missing from the input scan. By bridging the gap between raw geometric observations and functional decomposition, the proposed model can "hallucinate" hidden details with high structural fidelity.

Methodology: A Multi-Stage Pipeline

The authors propose a pipeline that combines structural reasoning with geometric priors to reconstruct complete part-based models from partial scans.

The multi-stage pipeline: from input scan to complete part decomposition using a Part Tree and geometric priors.

1. Object Detection and Orientation

The system first identifies objects within the 3D scene using a backbone network (MLCVNet). It predicts 3D bounding boxes and orientations, aligning objects to a canonical "up" direction. This alignment is a prerequisite for consistent part reasoning.

2. Semantic Part Decomposition (The Part Tree)

Rather than directly generating voxels, the model first predicts a Part Tree (or Part Graph). Using a Message-Passing Neural Network (GNN), the system determines which parts should exist (e.g., identifying that a chair requires four legs and a back) and establishes their relationships. This low-dimensional structural step allows the model to reason about missing components before attempting to generate complex geometry.

3. Prior-Guided Geometry

To generate the 3D masks for these parts, the model leverages geometric priors. The researchers pre-computed a library of "average" part shapes using $k$-means clustering on training data. The network learns to select and blend these priors to create a coarse 3D representation, effectively avoiding the need to reconstruct common shapes from scratch for every instance.

4. Residual Refinement

Finally, a 3D CNN performs residual refinement. It takes the coarse prior-based mask and the original partial scan to produce a high-resolution final part mask. By focusing on the "residual"—the difference between the generic prior and the specific object instance—the network can capture unique details while maintaining structural integrity.

Datasets and Benchmark

To train and evaluate the model, the authors created a novel benchmark by synthesizing three major datasets:

  • ScanNet: Real-world 3D indoor scans.
  • Scan2CAD: Alignments of clean CAD models to messy ScanNet objects.
  • PartNet: Fine-grained part annotations for the CAD models. This combination provides a ground truth where messy, incomplete inputs are paired with perfectly decomposed, complete part structures.

Results and Key Insights

The proposed method significantly outperforms existing baselines in both quantitative and qualitative measures. In terms of Intersection over Union (IoU), the model achieved 18.3%, a substantial improvement over the 13.3% achieved by the next best baseline (SG-NN + UNet).

Qualitative comparison of the proposed method against baselines and ground truth (GT). The model shows superior consistency and structural soundness.

Key insights from the study include:

  • Structure over Geometry: Using a Part Tree as an intermediary simplifies the high-dimensional problem of geometry generation into a more manageable structural problem.
  • Functional Hallucination: The model doesn't just segment visible points; it predicts what should be there based on learned priors. This allows for the reconstruction of hidden chair legs or the backs of cabinets pushed against walls.
  • Symmetry and Consistency: Qualitative results demonstrate that the model is much better at maintaining object symmetry compared to "blobby" reconstructions from baseline methods.

Limitations and Future Work

Despite its success, the model currently operates on a $32^3$ voxel grid, which limits its ability to capture very fine details like handles or thin wires. Furthermore, objects are currently processed independently. Future research could explore "relational reasoning," where the context of a room (e.g., a set of identical chairs around a table) informs the reconstruction of individual objects. Finally, moving toward more efficient representations like sparse voxels or point clouds could reduce the computational cost of generating dense volumetric masks for entire scenes.

Summary updated

Abstract

Recent advances in 3D semantic scene understanding have shown impressive progress in 3D instance segmentation, enabling object-level reasoning about 3D scenes; however, a finer-grained understanding is required to enable interactions with objects and their functional understanding. Thus, we propose the task of part-based scene understanding of real-world 3D environments: from an RGB-D scan of a scene, we detect objects, and for each object predict its decomposition into geometric part masks, which composed together form the complete geometry of the observed object. We leverage an intermediary part graph representation to enable robust completion as well as building of part priors, which we use to construct the final part mask predictions. Our experiments demonstrate that guiding part understanding through part graph to part prior-based predictions significantly outperforms alternative approaches to the task of part-based instance completion.
BibTeX
@inproceedings{bokhovkin2021towards,
  author = {Alexey Bokhovkin and Vladislav Ishimtsev and Emil Bogomolov and Denis Zorin and Alexey Artemov and Evgeny Burnaev and Angela Dai},
  title = {{Towards Part-Based Understanding of RGB-D Scans}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00740},
}
Figure