Cross-modal Learning for Image-Guided Point Cloud Shape Completion

Emanuele Aiello, Diego Valsesia, Enrico Magli

The Challenge of Incomplete 3D Data

3D sensors like LiDAR and depth cameras frequently capture incomplete point clouds due to occlusions, reflective surfaces, and limited viewing angles. Completing these partial shapes is a highly ill-posed inverse problem. While humans naturally use their 2D visual experience to infer 3D shapes, most computational approaches rely solely on 3D shape priors.

Recently, researchers have started using auxiliary 2D images to guide 3D completion. However, existing multimodal methods (such as the ViPC model) face a major bottleneck: they attempt to explicitly reconstruct a coarse 3D point cloud from the 2D image before fusing it with the partial 3D input. This intermediate step is an error-prone inverse problem itself. Furthermore, training these models typically requires massive datasets of complete 3D ground truth shapes, which are notoriously difficult and expensive to acquire in real-world scenarios.

To address these issues, researchers have introduced XMFnet (Cross-Modal Fusion network), a novel architecture that fuses modalities in a localized latent space and introduces a weakly-supervised training pipeline powered by a differentiable renderer.

The XMFnet Architecture

XMFnet tackles the fusion bottleneck by discarding the intermediate 3D reconstruction step entirely.

Overview of the XMFnet architecture featuring cross-modal attention and the differentiable rendering pipeline.
  1. Latent Space Modality Fusion: Instead of forcing the 2D image into a 3D point cloud, XMFnet extracts localized features from both modalities using specialized encoders (EdgeConv/SAGPool for 3D, ResNet18 for 2D). These features are then directly fused in a latent space using a sequence of Transformer-style cross-attention and self-attention layers. This allows the network to naturally find correspondences between 2D pixels and 3D regions.
  2. Attention-Based Decoder: The fused features are passed to a multi-branch decoder that predicts the coordinates of the missing points. These predicted points are then concatenated with the subsampled input point cloud to form the final completed shape.

Weakly-Supervised Training via Differentiable Rendering

To bypass the need for complete 3D ground truth, the authors introduce a novel weakly-supervised setting. If you don't have the full 3D shape, a 2D image from a different vantage point still contains vital clues about the occluded regions.

The weakly-supervised training pipeline utilizing a differentiable renderer to compute a 2D rendering loss.

The researchers pass the network's predicted 3D point cloud through a differentiable renderer (using alpha compositing) to generate a 2D silhouette. This silhouette is compared against the actual input image, providing a 2D supervisory signal for the missing 3D parts. By mathematically penalizing the 3D model if its generated shape doesn't "cast the right shadow" from the camera's perspective, the network learns to complete the shape. This rendering loss is combined with point cloud self-supervision techniques (resampling and mixup) using a Density-aware Chamfer Distance (DCD) loss.

Key Insights and Results

Trained and evaluated on the ShapeNet-ViPC Dataset (which contains over 38,000 objects paired with partial point clouds and 2D images), XMFnet demonstrates several critical advantages:

  • Direct feature fusion beats explicit reconstruction: Letting cross-attention figure out the 2D-to-3D mapping in a latent space is far more effective than forcing a network to guess a 3D shape from a 2D image before fusion.
  • State-of-the-Art Supervised Performance: In a fully supervised setting, XMFnet achieves a Mean Chamfer Distance (CD) of 1.443 ($\times 10^{-3}$), less than half the error of the previous best multimodal method, ViPC (3.308). It also vastly outperforms unimodal models like VRC-Net (4.598) and improved the Mean F-Score to 0.796 (up from ViPC's 0.591).
  • Powerful Weak Supervision: The weakly-supervised version of XMFnet (trained without complete 3D ground truth) achieved a CD of 2.426 on the 'airplane' class. Remarkably, this outperforms several fully supervised baselines (like TopNet and ECG), proving that 2D rendering loss can effectively substitute for missing 3D data.
  • The Importance of Viewpoint: The specific vantage point of the auxiliary image heavily impacts completion quality. Images that capture the occluded regions provide complementary information leading to massive improvements, while uninformative views yield results closer to 3D-only baselines.

Limitations and Future Work

Currently, XMFnet's evaluation is limited to synthetic data (ShapeNet). Real-world applications introduce complex domain shifts that the model hasn't yet been tested against, including sensor acquisition noise, cluttered backgrounds in the auxiliary 2D images, and unpredictable real-world occlusions. Future work will focus on extending this framework to handle real-world acquisitions and improving the architectural design to increase robustness against these environmental variables.

Summary updated

Abstract

In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel self-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significant improvements over state-of-the-art supervised methods for both unimodal and multimodal completion. We also show the effectiveness of the self-supervised approach which outperforms a number of supervised methods and is competitive with the latest supervised models only exploiting point cloud information.
BibTeX
@inproceedings{aiello2022cross,
  author = {Emanuele Aiello and Diego Valsesia and Enrico Magli},
  title = {{Cross-modal Learning for Image-Guided Point Cloud Shape Completion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=bi1BTcXa8Q},
}
Figure