IE-PMMA: Point Cloud Completion Through Inverse Edge-aware Upsampling and Precise Multi-Modal Feature Alignment

Ran Jia, Junpeng Xue, Shuai Ma, Wenbo Lu, Kelei WangIJCAI 2025 · DOI

TL;DR

This paper introduces IE-PMMA, a novel point cloud completion method that leverages edge information for the first time and improves multi-modal feature alignment. By precisely aligning 2D image and 3D point cloud features and introducing an "inverse edge-aware" upsampling strategy, IE-PMMA achieves state-of-the-art results in generating complete and dense 3D point clouds.

What Problem Does This Paper Address?

3D point cloud completion is a critical task in computer vision, aiming to reconstruct complete 3D shapes from sparse and incomplete point cloud data acquired by scanners. While multi-modal approaches, combining 2D images with point clouds, have shown promise, existing methods often struggle with accurately aligning features from these different modalities. Furthermore, the rich geometric details present in edge points of point clouds are largely unexplored in completion tasks. This paper addresses these limitations by proposing a method that enhances both multi-modal feature alignment and incorporates edge information for more robust and accurate point cloud completion.

Methods

IE-PMMA employs a two-stage framework: seed point generation and seed point upsampling.

1. Seed Points Generation with Precise Multi-Modal Feature Alignment (PMMA Module): This module focuses on accurately fusing 2D image and 3D point cloud features. It introduces a local-to-global approach:

  • Local Alignment: For each point in the 3D cloud, its corresponding 2D image features are extracted using camera projection and bilinear interpolation. This is done across multiple viewpoints to capture a comprehensive local view. A scaling factor is applied to align feature magnitudes.
  • Global Fusion: 3D features are extracted using PointNet++. These, along with the aligned 2D features, are then fused using a multi-head self-attention mechanism to achieve global feature alignment. The resulting global feature is used to regress the coordinates of seed points.

2. Edge Points Extraction Based on Edge Probability Volume (EPV Module): To incorporate edge information, the authors propose the EPV module. Instead of simple binary classification of edge points, EPV predicts the probability of a voxel being an edge voxel. This probabilistic approach is more robust to noise.

  • A simplified PointNet++ and 3D CNN extract feature grids.
  • Two decoders predict voxel edge probability and point coordinates within the voxel.
  • The ground truth probability is derived from the distance to known edge points, inspired by the first law of geography.
  • This module allows for extracting a fixed number of edge points, making it suitable for downstream tasks.

3. Seed Points Upsampling Based on Inverse Edge-Aware Upsampler (IEup Module): This is the core novelty for the upsampling stage.

  • Missing Importance (MI): Calculated as the chamfer distance between seed points and input partial points. Higher MI means the point is more "missing."
  • Inverse Edge Importance (IEI): Calculated as the chamfer distance between seed points and extracted edge points. Crucially, higher IEI means the point is further from an edge, implying it's more important for defining the global shape.
  • Aware Importance (AI): A combination of MI and IEI, weighted to prioritize global shape coherence.
  • A Missing & Inverse Edge-aware Self-Attention Module uses AI and partial features to generate aware features, which are then used with decoders to predict the displacement of seed points, leading to a coarse-to-fine upsampling process.

Datasets

The method was evaluated on three common datasets:

  • PCN: A subset of ShapeNet, containing 8 object categories.
  • ShapeNet34/55: Used to test generalization, with ShapeNet55 containing 55 categories and ShapeNet34 containing 34 visible and 21 invisible categories.
  • KITTI: A real-world dataset, specifically focusing on car objects, where ground truth is not available.

Key Insights and Ideas

  • Edge Information is Underexplored: The paper highlights that edge points, rich in local geometric details, have been largely ignored in point cloud completion, despite their utility in other 3D tasks.
  • Inverse Edge Importance for Global Shape: Counter-intuitively, assigning higher importance to points further from edges (Inverse Edge Importance) proves more effective for completion. This strategy guides the upsampling to focus on reconstructing the overall global shape rather than getting lost in local details.
  • Probabilistic Edge Prediction for Robustness: Moving from binary edge classification to probabilistic voxel prediction (EPV) makes the edge extraction more robust to noise and allows for a controllable number of edge points.
  • Precise Local-to-Global Multi-Modal Alignment: The PMMA module's combination of local point-to-pixel feature alignment and global attention-based fusion significantly improves how 2D and 3D information is integrated.

Key Results

IE-PMMA achieves state-of-the-art (SOTA) performance on the PCN and ShapeNet34/55 datasets across various metrics, including Chamfer Distance (CD), Density Chamfer Distance (DCD), and F1 Score.

  • On the PCN dataset, IE-PMMA shows improvements, particularly in DCD.
  • On ShapeNet34/55, it outperforms existing methods, demonstrating strong generalization.
  • On the KITTI dataset, it achieves SOTA in Fidelity.
  • Qualitative results show IE-PMMA generates more complete and visually coherent point clouds, accurately recovering local structures and global shapes. Ablation studies confirm the effectiveness of both the PMMA and IEup modules.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the two-step alignment and EPV module introduce additional computational costs and parameters. While justified by performance gains, this is a trade-off.
  • Future Work: The paper implicitly suggests that further exploration of edge-aware techniques in point cloud processing could be beneficial. The probabilistic nature of EPV also opens avenues for more sophisticated edge representations. The precise multi-modal alignment could be extended to other tasks requiring cross-modal understanding.

Summary updated

BibTeX
@inproceedings{jia2025ie,
  author = {Ran Jia and Junpeng Xue and Shuai Ma and Wenbo Lu and Kelei Wang},
  title = {{IE-PMMA: Point Cloud Completion Through Inverse Edge-aware Upsampling and Precise Multi-Modal Feature Alignment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/138},
}
Figure