TL;DR
This paper introduces "semantic instance completion," the task of inferring the complete 3D geometry of individual objects from incomplete scans. Their proposed RevealNet, a unified 3D neural network, jointly detects objects and predicts their full geometry, significantly outperforming existing methods and enabling more robust robotic interactions and scene understanding.
What Problem Does This Paper Address?
When we scan 3D environments using devices like RGB-D cameras, we often end up with incomplete data. This is because objects can be occluded, meaning we only see parts of them. This missing geometry is a major hurdle for many applications. For instance, a robot trying to grasp an object needs to know its complete shape, not just the visible portion. Similarly, for virtual reality or scene editing, having complete object models is crucial. Existing methods either focus on segmenting visible objects or completing entire scenes, but not on inferring the complete geometry of individual objects within a scene. This paper aims to bridge this gap by introducing the task of "semantic instance completion."
Methods
The core of this work is RevealNet, a novel end-to-end 3D convolutional neural network. It takes an RGB-D scan as input, which is represented as a Truncated Signed Distance Field (TSDF) in a volumetric grid. To leverage both visual and geometric information, color features from 2D RGB images are extracted and then back-projected into the 3D volume, where they are fused with the TSDF features.
This combined feature representation is fed into a unified encoder-decoder backbone. This backbone is designed to learn representations that inform both object detection and geometric completion. The network first performs object detection, predicting 3D bounding boxes and class labels for each object. Crucially, these predicted bounding boxes are then used to crop features from the backbone, which are then fed into a completion head to predict the per-voxel occupancy of the complete object geometry, including unseen parts.
A key innovation is the joint training of detection and completion. This allows information to flow from the completion task back to the detection parameters, improving both. The authors also introduce a global geometric completion loss as a proxy to further enhance the learning process.
Datasets
The authors evaluate their approach on two primary datasets:
- ScanNet: A real-world dataset of indoor scenes with rich annotations. They leverage Scan2CAD annotations for ground truth object locations and geometry.
- SUNCG: A synthetic dataset of indoor scenes, also used for training and evaluation.
They also introduce a new benchmark for semantic instance completion on ScanNet, using Scan2CAD annotations.
Key Insights and Ideas
- "Seeing behind objects" is a solvable problem: The paper demonstrates that it's possible to infer the complete geometry of objects even from partial scans.
- Joint detection and completion is synergistic: By training object detection and geometric completion together, each task benefits the other. Completion helps detection by providing a more complete understanding of object shapes, and detection provides the necessary localization for completion.
- Instance-level completion is crucial for interaction: Unlike scene completion, focusing on individual objects unlocks more granular control and understanding for applications like robotics.
- Color information significantly helps: Integrating RGB image features alongside geometric data leads to substantial performance improvements.
Key Results
RevealNet significantly outperforms state-of-the-art methods on the semantic instance completion task:
- ScanNet (real-world): Over 15% improvement in [email protected] compared to decoupled approaches.
- SUNCG (synthetic): Over 18% improvement in [email protected].
- The paper also shows that predicting complete object geometry improves 3D detection and instance segmentation performance, with gains of up to ~12% in 3D detection and ~6% in instance segmentation on ScanNet.
- Qualitative results (Figures 3 and 4) show RevealNet successfully reconstructing complete object geometries, including missing parts like chair legs, even in highly partial scans.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Binary mask output: The current output is a binary mask, which might limit the representation of fine geometric details. Future work could explore more detailed 3D representations like distance fields.
- Axis-aligned bounding boxes: The object detection uses axis-aligned bounding boxes. Predicting object orientation would be a valuable addition.
- Static scenes: The current approach is designed for static environments. Extending it to handle dynamic scenes with moving objects presents an interesting avenue for future research.
Summary updated