Introduction
Reconstructing a complete 3D shape from a single viewpoint, such as a depth camera, is a fundamental challenge in computer vision. Because sensors only capture the "front" of an object, the system must infer the geometry of occluded regions. While most man-made objects exhibit reflection symmetry, leveraging this property is a "chicken and egg" problem: accurate symmetry detection requires a complete shape, yet accurate shape completion often depends on knowing the symmetry plane.
Previous approaches typically assumed a perfect symmetry plane or relied on fixed viewpoints, which often fail when faced with awkward input angles or complex occlusions. The Iterative Symmetry Completion Network (ISCNet) addresses this by treating symmetry detection and shape completion as complementary, mutually reinforcing tasks.
Methodology
ISCNet breaks the cycle of dependency between symmetry and completion through an iterative refinement framework and intelligent viewpoint selection.
Iterative Refinement Loop
The process begins with a partial point cloud derived from a single depth image. The system estimates an initial symmetry plane, uses it to guide shape completion, and then utilizes the newly completed geometry to re-estimate a more accurate symmetry plane. This refinement cycle typically occurs over 20 iterations, progressively sharpening both the symmetry alignment and the final 3D model.
RL-based Viewpoint Selection and 2D Inpainting
Rather than relying on a simple front-to-back flip, ISCNet employs a Reinforcement Learning (RL) agent to identify optimal pairs of symmetric viewpoints. This agent learns to "look" at the partial shape from angles that maximize the discovery of missing geometry.
To maintain high resolution without the computational cost of dense 3D voxels, ISCNet projects 3D data into 2D depth and normal maps. A specialized inpainting module, utilizing Partial Convolutions, fills the gaps in these 2D representations. These repaired maps are then projected back into 3D space to update the evolving point cloud.
Experimental Evaluation
The model was trained and tested on the ShapeNet dataset across eight categories (including chairs, cars, and lamps) and validated on the real-world ScanNet dataset using Scan2CAD ground truths.
Key Results
ISCNet achieved state-of-the-art performance on both synthetic and real-world data:
- Quantitative Performance: On ShapeNet, ISCNet reached a Chamfer Distance (CD) of 1.48, significantly outperforming previous leaders like Front2Back (2.36) and PoinTr (2.48). On real-world ScanNet data, it reduced the error metric to 4.19, compared to 5.47 for the next best method, VRCNet.
- Qualitative Superiority: Visually, ISCNet produces cleaner surfaces and excels at preserving thin structures, such as chair legs or lamp stands, which are often blurred or lost in voxel-based or point-cloud-only methods.
Conclusion and Future Work
ISCNet demonstrates that symmetry is most effective when used as a core structural guide throughout the reconstruction process rather than a post-processing step. By converting 3D completion into a 2D inpainting task guided by RL, the model achieves high-resolution results that generalize well to noisy real-world data.
However, the iterative nature of the pipeline currently presents a computational bottleneck, requiring approximately 81 seconds per shape. Future research will focus on optimizing the pipeline for real-time applications and extending the framework to handle objects with multiple symmetry planes or complex rotational symmetries.
Summary updated