Abstract
Point cloud completion is a critical task in 3D computer vision, aiming to recover complete shapes from partial scans. Traditional methods often struggle with fine geometric details due to the inherent sparsity and occlusion in 3D data. We present CDPNet (Cross-Modal Dual Phases Network), a novel framework that leverages 2D image information to guide 3D reconstruction. By employing a "divide-and-conquer" patch-based strategy and a style-based feature fusion module, CDPNet achieves state-of-the-art performance on the ShapeNet-ViPC dataset, significantly improving both quantitative metrics and qualitative fidelity.
Introduction
In real-world applications such as autonomous driving and robotics, 3D sensors like LiDAR frequently produce incomplete "partial" point clouds. These scans often represent only the visible "shell" of an object, missing critical structural information due to occlusions or sensor limitations. While standard encoder-decoder architectures attempt to infer missing parts using learned shape priors, they frequently fail to reconstruct intricate details, resulting in blurred or incomplete geometries.
CDPNet addresses this by integrating 2D visual experience into the 3D reconstruction process. Humans naturally use 2D visual cues to infer 3D structures; similarly, CDPNet utilizes a single-view 2D image of the target object to provide global structural guidance, filling the semantic gaps that 3D-only sensors might overlook.
Methodology
The CDPNet architecture is structured into two distinct phases, as illustrated in the detailed framework overview.
Phase I: Coarse Reconstruction and Patching
In the first phase, the network takes a single-view 2D image $I$ and reconstructs a coarse 3D point cloud $C$. This coarse cloud is then segmented into $M$ distinct patches. This "divide-and-conquer" approach allows the model to focus on local geometric details, preventing the information loss that occurs when a single global latent code is used to describe a complex 3D object.
Phase II: Dense Generation and Feature Fusion
Phase II focuses on generating a dense, high-fidelity point cloud by fusing features from both modalities:
- Feature Fusion Module: A cross-attention mechanism treats image features as queries and point-wise features from the partial scan (extracted via DGCNN) as keys and values. This process yields a "Style Code" that encapsulates the global structural essence of the object.
- Multi-Patch Generator: Inspired by generative adversarial networks, the architecture uses Adaptive Instance Normalization (AdaIN) to inject the Style Code into the generation of local patches. This ensures that while each patch captures local geometric nuances, they collectively adhere to a consistent global structure defined by the 2D image.
Experiments and Results
The performance of CDPNet was evaluated on the ShapeNet-ViPC dataset, which provides aligned 2D images and 3D point clouds across eight object categories.
Quantitative Performance
CDPNet achieved state-of-the-art results across all categories. It recorded an average Chamfer Distance (CD) of 1.066, a significant improvement over the previous leading cross-modal method, XMFNet (1.367). Additionally, it achieved an average F-Score of 0.869, surpassing XMFNet’s 0.810.
Qualitative Comparison
Visually, CDPNet demonstrates a superior ability to reconstruct complex and thin structures. As shown in the qualitative comparisons, CDPNet accurately recovers features like chair legs and specific vehicle contours that other methods often over-smooth or omit.
Conclusion
CDPNet introduces a powerful synergy between 2D visual cues and 3D geometric constraints. By treating 2D information as a "style" blueprint and employing a patch-based generation strategy, it effectively overcomes the limitations of single-modality completion. While the model's performance is somewhat dependent on the input image's viewpoint, it consistently outperforms 3D-only baselines. Future work will explore view-selection optimization and improving the seamlessness of patch boundaries.
Summary updated