TL;DR
This paper compares two neural network approaches, DeepSDF and MirrorNet, for reconstructing the complete 3D shape of objects from a single RGB-D camera view, specifically focusing on filling in occluded or non-visible surfaces. MirrorNet, a view-dependent method, demonstrates significantly faster inference times and more accurate reconstructions, particularly in terms of Hausdorff distance, making it more suitable for real-time robotics applications.
What Problem Does This Paper Address?
Robots often rely on RGB-D cameras to perceive their environment, but a single camera view provides incomplete information due to occlusions. This limits their ability to perform tasks like grasping or manipulation, which require a full understanding of an object's geometry. While various 3D reconstruction techniques exist, many struggle with resource consumption or are not optimized for real-time robotic scenarios. This paper addresses the challenge of reconstructing the entire 3D shape of an object, including its hidden parts, from a single, potentially noisy, RGB-D input, and compares two distinct neural network-based strategies for this task.
Methods
The paper compares two main methods:
- DeepSDF: This method represents objects using a Signed Distance Transform (SDF) learned by a fully-connected neural network. It operates directly in 3D space, predicting the distance from any given point to the object's surface. A key requirement for DeepSDF is that it needs depth observations in a canonical shape frame of reference.
- MirrorNet: This is a view-dependent approach that reconstructs occluded parts by generating new images from the "other side" of the observed object. It utilizes the input depth image and projects the point cloud to a virtual camera position to generate new depth images, which are then processed by a Convolutional Neural Network. MirrorNet does not have the canonical frame requirement and can directly use noisy depth images from a real robot.
Datasets
- ShapeNet: The authors used the ShapeNet dataset to train and evaluate their models. They selected 6 representative categories of graspable entities and used 30 instances per category for training. For testing, they used another 10 instances from the same categories.
Key Insights and Ideas
- View-dependent vs. View-independent: The study highlights a crucial trade-off between view-dependent methods (like MirrorNet) that generate new views and view-independent methods (like DeepSDF) that operate in a global 3D space.
- Speed vs. Accuracy Trade-off: MirrorNet, despite its view-dependent nature, achieves superior speed and accuracy in many metrics, suggesting that generating intermediate views can be more efficient than direct 3D space sampling for certain tasks.
- Robustness to Noise: MirrorNet's ability to directly use noisy depth camera images from a real robot is a significant advantage for practical robotics applications.
- Limitations of SDF for Incomplete Data: DeepSDF struggled with calculating SDF values for incomplete object meshes from a single view, requiring specific thresholds and preprocessing to mitigate issues.
Key Results
- Speed: MirrorNet is significantly faster, with an inference time of 22 ms per object, compared to DeepSDF's 15000 ms (15 seconds) per object.
- Accuracy (Chamfer Distance): Both systems returned similar results for the Chamfer distance ($d_c$).
- Accuracy (Hausdorff Distance): MirrorNet's Hausdorff distance ($d_h$) was 2-3 times better than DeepSDF's, indicating a more accurate reconstruction of the overall shape, especially for finer details.
- Qualitative Results: MirrorNet produced visually better reconstructions in some cases, though it generated fewer points. DeepSDF sometimes failed to reconstruct specific features like the handle of a mug or the rounded shape of a can. MirrorNet also generated sparse points that could be filtered out.
Limitations and Future Work
- MirrorNet's Sparse Points: MirrorNet generates random sparse points between the reconstructed surface and the camera pose, which require post-processing (e.g., voxel-based filters) to remove.
- DeepSDF's Canonical Frame Requirement: DeepSDF's need for depth observations in a canonical shape frame of reference can be a limitation in real-world scenarios where such alignment might not be readily available.
- Future Work: The authors plan to extract features from the view-dependent models (MirrorNet) and use them for efficient grasping and manipulation of objects from a single camera view.
Summary updated