Abstract
In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significant improvements over state-of-the-art supervised methods for both unimodal and multimodal completion. We also show the effectiveness of the weakly-supervised approach which outperforms a number of supervised methods and is competitive with the latest supervised models only exploiting point cloud information.
Introduction
3D sensing technologies like LiDAR and depth cameras are essential for autonomous systems, yet they frequently produce incomplete "point clouds" due to occlusions, reflections, or sensor limitations. Completing these shapes is an ill-posed problem, as multiple valid geometries can correspond to the same partial input. While humans naturally leverage 2D visual context to infer 3D structure, effectively fusing these modalities in deep learning has proven challenging. Previous approaches often attempted to reconstruct a full 3D model from the 2D image first, creating a bottleneck where errors in the initial reconstruction propagated to the final output. Furthermore, the reliance on high-fidelity 3D ground truth for training limits real-world applicability.
The XMFnet Architecture
The proposed XMFnet (Cross-Modal Fusion network) addresses these challenges by performing direct feature fusion in a localized latent space. Instead of intermediate 3D reconstruction, XMFnet extracts localized features from the point cloud and a paired 2D image using specialized encoders.
The architecture employs a Point Cloud Encoder based on graph convolutions and a ResNet-18 Image Encoder. These features are integrated using Transformer-style cross-attention, allowing the model to query the image for specific details relevant to particular 3D regions. This approach avoids the reconstruction bottleneck and allows the model to "borrow" missing details directly from the 2D modality. The researchers also found that viewpoint complementarity is critical: images taken from angles that reveal occluded regions of the point cloud provide significantly more useful information for completion than those taken from the same perspective as the 3D sensor.
Weakly-Supervised Learning via Differentiable Rendering
A key innovation of XMFnet is its ability to learn in a weakly-supervised setting. When 3D ground truth is unavailable, the model utilizes a differentiable renderer to project the predicted 3D completion back into a 2D silhouette.
By comparing this projected silhouette to the silhouette of the input image, the network receives a supervisory signal based on "reprojection error." If the 3D shape is inaccurate, its 2D projection will not match the observed image. To handle the lack of depth information in silhouettes, the authors utilize the Density-aware Chamfer Distance (DCD), which helps mitigate ambiguities during training by focusing on point distribution fidelity.
Experimental Evaluation
The model was evaluated on the ShapeNet-ViPC dataset, which contains over 38,000 objects across 13 categories. Each object is represented by 24 partial point clouds and corresponding 2D images from varying viewpoints.
Key Results
- Supervised Performance: XMFnet achieved a Mean Chamfer Distance (CD) of 1.443, significantly outperforming the previous state-of-the-art (ViPC at 3.308).
- Weakly-Supervised Efficacy: Even without 3D ground truth, the weakly-supervised XMFnet outperformed several fully-supervised baselines, such as PCN and TopNet.
- Qualitative Quality: Visual inspections confirm that XMFnet produces cleaner completions with more uniform point distributions and fewer outliers than competing methods.
Conclusion and Future Work
XMFnet demonstrates that direct cross-modal fusion and differentiable rendering are powerful tools for 3D shape completion. However, several challenges remain. The current reliance on synthetic datasets like ShapeNet may not fully capture the noise and complexity of real-world environments. Additionally, the weakly-supervised approach requires known camera parameters, which may need estimation in uncontrolled settings. Future research could explore more sophisticated rendering techniques or depth-map supervision to further reduce ambiguity in the learning process.
Summary updated