Introduction
While 3D scene understanding has made significant progress in object-level identification and instance segmentation, most existing methods treat objects as monolithic entities. However, human interaction with the physical world is fundamentally part-based: we sit on a chair's seat, pull a cabinet's handle, or place objects on a table's top. Understanding these functional components is critical for robotics and augmented reality.
Real-world RGB-D scans (e.g., from Kinect or mobile sensors) are often noisy, cluttered, and incomplete due to occlusions. This paper introduces the task of semantic part completion, which aims to predict the full, fine-grained part structure of objects even when significant portions are missing from the input scan. By bridging the gap between raw geometric observations and functional decomposition, the proposed model can "hallucinate" hidden details with high structural fidelity.
Methodology: A Multi-Stage Pipeline
The authors propose a pipeline that combines structural reasoning with geometric priors to reconstruct complete part-based models from partial scans.
1. Object Detection and Orientation
The system first identifies objects within the 3D scene using a backbone network (MLCVNet). It predicts 3D bounding boxes and orientations, aligning objects to a canonical "up" direction. This alignment is a prerequisite for consistent part reasoning.
2. Semantic Part Decomposition (The Part Tree)
Rather than directly generating voxels, the model first predicts a Part Tree (or Part Graph). Using a Message-Passing Neural Network (GNN), the system determines which parts should exist (e.g., identifying that a chair requires four legs and a back) and establishes their relationships. This low-dimensional structural step allows the model to reason about missing components before attempting to generate complex geometry.
3. Prior-Guided Geometry
To generate the 3D masks for these parts, the model leverages geometric priors. The researchers pre-computed a library of "average" part shapes using $k$-means clustering on training data. The network learns to select and blend these priors to create a coarse 3D representation, effectively avoiding the need to reconstruct common shapes from scratch for every instance.
4. Residual Refinement
Finally, a 3D CNN performs residual refinement. It takes the coarse prior-based mask and the original partial scan to produce a high-resolution final part mask. By focusing on the "residual"—the difference between the generic prior and the specific object instance—the network can capture unique details while maintaining structural integrity.
Datasets and Benchmark
To train and evaluate the model, the authors created a novel benchmark by synthesizing three major datasets:
- ScanNet: Real-world 3D indoor scans.
- Scan2CAD: Alignments of clean CAD models to messy ScanNet objects.
- PartNet: Fine-grained part annotations for the CAD models. This combination provides a ground truth where messy, incomplete inputs are paired with perfectly decomposed, complete part structures.
Results and Key Insights
The proposed method significantly outperforms existing baselines in both quantitative and qualitative measures. In terms of Intersection over Union (IoU), the model achieved 18.3%, a substantial improvement over the 13.3% achieved by the next best baseline (SG-NN + UNet).
Key insights from the study include:
- Structure over Geometry: Using a Part Tree as an intermediary simplifies the high-dimensional problem of geometry generation into a more manageable structural problem.
- Functional Hallucination: The model doesn't just segment visible points; it predicts what should be there based on learned priors. This allows for the reconstruction of hidden chair legs or the backs of cabinets pushed against walls.
- Symmetry and Consistency: Qualitative results demonstrate that the model is much better at maintaining object symmetry compared to "blobby" reconstructions from baseline methods.
Limitations and Future Work
Despite its success, the model currently operates on a $32^3$ voxel grid, which limits its ability to capture very fine details like handles or thin wires. Furthermore, objects are currently processed independently. Future research could explore "relational reasoning," where the context of a room (e.g., a set of identical chairs around a table) informs the reconstruction of individual objects. Finally, moving toward more efficient representations like sparse voxels or point clouds could reduce the computational cost of generating dense volumetric masks for entire scenes.
Summary updated