3D Reconstruction of non-visible surfaces of objects from a Single Depth View - Comparative Study

Rafal Staszak, Piotr Michalek, Jakub Chudzinski, Marek Sewer Kopicki, Dominik BelterarXiv 2025 · DOI · arXiv

TL;DR

This paper compares two neural network approaches, DeepSDF and MirrorNet, for reconstructing the complete 3D shape of objects from a single RGB-D camera view, specifically focusing on filling in occluded or non-visible surfaces. MirrorNet, a view-dependent method, demonstrates significantly faster inference times and more accurate reconstructions, particularly in terms of Hausdorff distance, making it more suitable for real-time robotics applications.

What Problem Does This Paper Address?

Robots often rely on RGB-D cameras to perceive their environment, but a single camera view provides incomplete information due to occlusions. This limits their ability to perform tasks like grasping or manipulation, which require a full understanding of an object's geometry. While various 3D reconstruction techniques exist, many struggle with resource consumption or are not optimized for real-time robotic scenarios. This paper addresses the challenge of reconstructing the entire 3D shape of an object, including its hidden parts, from a single, potentially noisy, RGB-D input, and compares two distinct neural network-based strategies for this task.

Methods

The paper compares two main methods:

  • DeepSDF: This method represents objects using a Signed Distance Transform (SDF) learned by a fully-connected neural network. It operates directly in 3D space, predicting the distance from any given point to the object's surface. A key requirement for DeepSDF is that it needs depth observations in a canonical shape frame of reference.
  • MirrorNet: This is a view-dependent approach that reconstructs occluded parts by generating new images from the "other side" of the observed object. It utilizes the input depth image and projects the point cloud to a virtual camera position to generate new depth images, which are then processed by a Convolutional Neural Network. MirrorNet does not have the canonical frame requirement and can directly use noisy depth images from a real robot.

Datasets

  • ShapeNet: The authors used the ShapeNet dataset to train and evaluate their models. They selected 6 representative categories of graspable entities and used 30 instances per category for training. For testing, they used another 10 instances from the same categories.

Key Insights and Ideas

  • View-dependent vs. View-independent: The study highlights a crucial trade-off between view-dependent methods (like MirrorNet) that generate new views and view-independent methods (like DeepSDF) that operate in a global 3D space.
  • Speed vs. Accuracy Trade-off: MirrorNet, despite its view-dependent nature, achieves superior speed and accuracy in many metrics, suggesting that generating intermediate views can be more efficient than direct 3D space sampling for certain tasks.
  • Robustness to Noise: MirrorNet's ability to directly use noisy depth camera images from a real robot is a significant advantage for practical robotics applications.
  • Limitations of SDF for Incomplete Data: DeepSDF struggled with calculating SDF values for incomplete object meshes from a single view, requiring specific thresholds and preprocessing to mitigate issues.

Key Results

  • Speed: MirrorNet is significantly faster, with an inference time of 22 ms per object, compared to DeepSDF's 15000 ms (15 seconds) per object.
  • Accuracy (Chamfer Distance): Both systems returned similar results for the Chamfer distance ($d_c$).
  • Accuracy (Hausdorff Distance): MirrorNet's Hausdorff distance ($d_h$) was 2-3 times better than DeepSDF's, indicating a more accurate reconstruction of the overall shape, especially for finer details.
  • Qualitative Results: MirrorNet produced visually better reconstructions in some cases, though it generated fewer points. DeepSDF sometimes failed to reconstruct specific features like the handle of a mug or the rounded shape of a can. MirrorNet also generated sparse points that could be filtered out.

Limitations and Future Work

  • MirrorNet's Sparse Points: MirrorNet generates random sparse points between the reconstructed surface and the camera pose, which require post-processing (e.g., voxel-based filters) to remove.
  • DeepSDF's Canonical Frame Requirement: DeepSDF's need for depth observations in a canonical shape frame of reference can be a limitation in real-world scenarios where such alignment might not be readily available.
  • Future Work: The authors plan to extract features from the view-dependent models (MirrorNet) and use them for efficient grasping and manipulation of objects from a single camera view.

Summary updated

BibTeX
@misc{staszak2025d,
  author = {Rafal Staszak and Piotr Michalek and Jakub Chudzinski and Marek Sewer Kopicki and Dominik Belter},
  title = {{3D Reconstruction of non-visible surfaces of objects from a Single Depth View - Comparative Study}},
  year = {2025},
  eprint = {2501.16101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.16101},
}
Figure