Abstract
During 3D reconstruction, it is often the case that people cannot scan each individual object from all views, resulting in missing geometry in the captured scan. This missing geometry can be fundamentally limiting for many applications, e.g., a robot needs to know the unseen geometry to perform a precise grasp on an object. Thus, we introduce the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances and infer their complete object geometry. This will open up new possibilities for interactions with objects in a scene, for instance for virtual or robotic agents. We tackle this problem by introducing RevealNet, a new data-driven approach that jointly detects object instances and predicts their complete geometry. This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts. RevealNet is an end-to-end 3D neural network architecture that leverages joint color and geometry feature learning. The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass. We show that predicting complete object geometry improves both 3D detection and instance segmentation performance. We evaluate on both real and synthetic scan benchmark data for the new task, where we outperform state-of-the-art approaches by over 15 in [email protected] on ScanNet, and over 18 in [email protected] on SUNCG.
Introduction
Capturing 3D environments with RGB-D sensors like Kinect or LiDAR-equipped smartphones is a standard practice in computer vision. However, these scans are inherently incomplete. Occlusions prevent cameras from seeing the backs of chairs, the floor under tables, or the interiors of cabinets. For a robot attempting to navigate or interact with these objects, these "holes" represent a significant challenge. Traditional approaches treat this as two separate problems: identifying visible segments (instance segmentation) or filling in holes globally (scan completion). This paper proposes Semantic Instance Completion (SIC), a unified task to detect and complete individual object instances simultaneously.
RevealNet: Methods and Architecture
The core contribution is RevealNet, an end-to-end 3D convolutional neural network designed to "see behind" occlusions. RevealNet takes a partial 3D scan, represented as a Truncated Signed Distance Field (TSDF), along with corresponding color images.
The architecture employs several key innovations:
- Multi-Modal Feature Fusion: By back-projecting 2D color features into the 3D volumetric grid, the model leverages visual cues like texture to disambiguate geometry.
- 3D Region Proposal Network (RPN): The network identifies potential object locations and generates 3D bounding boxes.
- Joint Completion Head: Within each proposed box, the network predicts a per-voxel occupancy mask. Unlike standard segmentation, this mask aims to represent the entire object, including unobserved parts.
- Global Proxy Loss: A secondary loss function forces the backbone to predict the completion of the entire scene, improving the quality of learned features.
Evaluation and Results
RevealNet was benchmarked on ScanNet (real-world scans aligned with CAD models via Scan2CAD) and SUNCG (synthetic environments). The results demonstrate a significant performance leap over decoupled baselines. On ScanNet, RevealNet achieved 21.77 [email protected], compared to just ~5.2 mAP for sequential methods.
Key insights from the evaluation include:
- Completion Aids Detection: Forcing the model to predict the full geometry of an object makes it more robust at detecting that object from partial views.
- Importance of Color: Ablation studies showed that removing color features leads to a massive drop in performance, as color provides vital semantic context that geometry alone cannot provide.
- End-to-End Synergy: The unified approach outperforms pipelines that separate detection and completion, proving that information flow between these tasks is critical.
Conclusion
RevealNet represents a major step toward semantically meaningful 3D scene understanding. While current outputs are voxel-based and somewhat low-resolution, the ability to hallucinate hidden geometry opens new doors for robotics and AR. Future work will focus on increasing geometric fidelity and handling dynamic environments.
Summary updated