Single Depth-image 3D Reflection Symmetry and Shape Prediction

Zhaoxuan Zhang, Bo Dong, Tong Li, Felix Heide, Pieter Peers, Baocai Yin, Xin Yang

Introduction

Reconstructing a complete 3D shape from a single viewpoint, such as a depth camera, is a fundamental challenge in computer vision. Because sensors only capture the "front" of an object, the system must infer the geometry of occluded regions. While most man-made objects exhibit reflection symmetry, leveraging this property is a "chicken and egg" problem: accurate symmetry detection requires a complete shape, yet accurate shape completion often depends on knowing the symmetry plane.

Previous approaches typically assumed a perfect symmetry plane or relied on fixed viewpoints, which often fail when faced with awkward input angles or complex occlusions. The Iterative Symmetry Completion Network (ISCNet) addresses this by treating symmetry detection and shape completion as complementary, mutually reinforcing tasks.

Methodology

ISCNet breaks the cycle of dependency between symmetry and completion through an iterative refinement framework and intelligent viewpoint selection.

Iterative Refinement Loop

The process begins with a partial point cloud derived from a single depth image. The system estimates an initial symmetry plane, uses it to guide shape completion, and then utilizes the newly completed geometry to re-estimate a more accurate symmetry plane. This refinement cycle typically occurs over 20 iterations, progressively sharpening both the symmetry alignment and the final 3D model.

Input depth map and observed surface representation used for symmetry estimation.

RL-based Viewpoint Selection and 2D Inpainting

Rather than relying on a simple front-to-back flip, ISCNet employs a Reinforcement Learning (RL) agent to identify optimal pairs of symmetric viewpoints. This agent learns to "look" at the partial shape from angles that maximize the discovery of missing geometry.

To maintain high resolution without the computational cost of dense 3D voxels, ISCNet projects 3D data into 2D depth and normal maps. A specialized inpainting module, utilizing Partial Convolutions, fills the gaps in these 2D representations. These repaired maps are then projected back into 3D space to update the evolving point cloud.

Experimental Evaluation

The model was trained and tested on the ShapeNet dataset across eight categories (including chairs, cars, and lamps) and validated on the real-world ScanNet dataset using Scan2CAD ground truths.

Key Results

ISCNet achieved state-of-the-art performance on both synthetic and real-world data:

  • Quantitative Performance: On ShapeNet, ISCNet reached a Chamfer Distance (CD) of 1.48, significantly outperforming previous leaders like Front2Back (2.36) and PoinTr (2.48). On real-world ScanNet data, it reduced the error metric to 4.19, compared to 5.47 for the next best method, VRCNet.
  • Qualitative Superiority: Visually, ISCNet produces cleaner surfaces and excels at preserving thin structures, such as chair legs or lamp stands, which are often blurred or lost in voxel-based or point-cloud-only methods.
Qualitative comparison showing ISCNet (e) producing results closest to the Ground Truth (f) compared to existing methods like PoinTr and IF-Net.

Conclusion and Future Work

ISCNet demonstrates that symmetry is most effective when used as a core structural guide throughout the reconstruction process rather than a post-processing step. By converting 3D completion into a 2D inpainting task guided by RL, the model achieves high-resolution results that generalize well to noisy real-world data.

However, the iterative nature of the pipeline currently presents a computational bottleneck, requiring approximately 81 seconds per shape. Future research will focus on optimizing the pipeline for real-time applications and extending the framework to handle objects with multiple symmetry planes or complex rotational symmetries.

Summary updated

Abstract

In this paper, we present Iterative Symmetry Completion Network (ISCNet), a single depth-image shape completion method that exploits reflective symmetry cues to obtain more detailed shapes. The efficacy of single depth-image shape completion methods is often sensitive to the accuracy of the symmetry plane. ISCNet therefore jointly estimates the symmetry plane and shape completion iteratively; more complete shapes contribute to more robust symmetry plane estimates and vice versa. Furthermore, our shape completion method operates in the image domain, enabling more efficient high-resolution, detailed geometry reconstruction. We perform the shape completion from pairs of viewpoints, reflected across the symmetry plane, predicted by a reinforcement learning agent to improve robustness and to simultaneously explicitly leverage symmetry. We demonstrate the effectiveness of ISCNet on a variety of object categories on both synthetic and real-scanned datasets.
BibTeX
@inproceedings{zhang2023single,
  author = {Zhaoxuan Zhang and Bo Dong and Tong Li and Felix Heide and Pieter Peers and Baocai Yin and Xin Yang},
  title = {{Single Depth-image 3D Reflection Symmetry and Shape Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00817},
}
Figure