SPU-IMR: Self-supervised Arbitrary-scale Point Cloud Upsampling via Iterative Mask-recovery Network

Ziming Nie, Qiao Wu, Chenlei Lv, Siwen Quan, Zhaoshuai Qi, Muze Wang, Jiaqi YangAAAI 2025 · DOI

TL;DR

This paper introduces SPU-IMR, a novel self-supervised method for point cloud upsampling that treats the problem as a global shape completion task rather than local interpolation. By iteratively recovering masked patches of a point cloud, SPU-IMR achieves arbitrary-scale upsampling with a single training pass, outperforming existing methods in both quantitative and qualitative evaluations.

What Problem Does This Paper Address?

Point clouds, a crucial 3D representation, are often sparse and non-uniform due to the limitations of 3D scanning devices. This sparsity hinders their use in various downstream applications like autonomous driving, virtual reality, and 3D reconstruction, where dense and uniform point clouds are highly desirable. Existing methods primarily rely on local interpolation in either feature or point space. While effective to some extent, these approaches struggle with capturing complex local topological structures and variations. Furthermore, many state-of-the-art methods require extensive supervised training, which is challenging to apply in real-world scenarios where labeled data is scarce. This paper aims to address these limitations by proposing a self-supervised approach that can generate high-quality, dense point clouds at arbitrary scales without requiring large amounts of labeled data.

Methods

SPU-IMR frames point cloud upsampling as an iterative mask-recovery problem. The core idea is to divide the input point cloud into patches, mask some of them, and then use a neural network to reconstruct the missing patches based on the visible ones.

Here's a breakdown of the key components:

  • Local Patch Masking: The input point cloud is first segmented into patches using FPS (Farthest Point Sampling) and KNN (K-Nearest Neighbors). A predefined mask ratio determines which patches are masked, creating visible and masked sets. Points within each patch are then transformed into a local coordinate system for standardization.
  • Visible Feature Extraction: A dedicated encoder extracts features from the visible point cloud patches. This involves shared Multi-Layer Perceptrons (MLPs) and max-pooling for local and global features, followed by a Point Transformer to capture contextual information and a Transformer module for feature enhancement.
  • Iterative Deformation Block: This is the heart of the method. Learnable point cloud patches are initialized and then iteratively deformed. Guided by the extracted visible features and the positional encoding of center points, this block refines the generated patches to match the topological structure of the masked regions. The deformation process is layered, with each layer building upon the output of the previous one.
  • Self-supervised Completion: After the iterative deformation, a shared coordinate prediction module (a single MLP) outputs the deformed point cloud patches. The loss function used is the Earth Mover's Distance (EMD) between the predicted and ground truth masked patches. EMD is chosen over Chamfer Distance for its ability to prevent clustering and better align with topological structures.
  • Multi-Mask-Recovery (MMR) Module (for Testing): To achieve arbitrary-scale upsampling, the MMR module is employed during inference. It involves using multiple different mask sequences to generate various combinations of completed patches. These patches are then merged with the original input point cloud, followed by outlier removal and FPS to obtain the final upsampled point cloud at the desired density.

Datasets

The paper uses the dataset introduced by Zhao et al. (2022, 2023) for training and testing. This dataset consists of 4719 point clouds, split into 3672 training samples and 1047 testing samples. Each sample initially contains 2048 points and is further down-sampled to 1024 points for training and testing. The paper also mentions evaluating on the KITTI dataset for real-world scenarios.

Key Insights and Ideas

  • Shifting from Interpolation to Completion: The most significant insight is reframing point cloud upsampling as a global shape completion problem rather than a local interpolation task. This allows the model to learn more holistic shape structures.
  • Self-Supervised Learning for Upsampling: The paper demonstrates that effective point cloud upsampling can be achieved through self-supervision, reducing the reliance on large, labeled datasets. This is a crucial step towards more practical applications.
  • Iterative Mask-Recovery: The iterative nature of the mask-recovery process allows for a more robust and accurate reconstruction of missing point cloud regions, gradually refining the shape and topology.
  • Arbitrary-Scale Upsampling with MMR: The MMR module is a clever way to achieve flexible upsampling ratios at test time without retraining, by strategically combining results from different masking strategies.
  • EMD Loss for Topological Accuracy: The choice of EMD loss over Chamfer Distance is key for preserving and accurately reconstructing the local topological structures of the point clouds.

Key Results

  • Quantitative Performance: SPU-IMR achieves competitive results compared to state-of-the-art (SoTA) methods across various upsampling ratios (2x, 4x, 8x) and metrics (CD, EMD, HD, F-Score, P2F). Notably, it performs best on EMD, HD, and F-Score for 2x upsampling and achieves the best P2F for 4x upsampling. Even though it's self-supervised, its performance is comparable to or better than many supervised methods.
  • Uniformity: The method demonstrates superior uniformity, as indicated by higher NUC scores, producing more evenly distributed point clouds.
  • Qualitative Results: Visual comparisons show that SPU-IMR effectively restores local structures, eliminates noise, and produces more accurate and complete point clouds compared to other methods, particularly on complex objects like chairs and airplanes. It also shows good generalization on unseen models and real-world data (KITTI dataset).
  • Efficiency: The MMR module allows for arbitrary-scale upsampling with a single training pass.

Limitations and Future Work

  • Mask Ratio Sensitivity: The paper notes that performance is sensitive to the mask ratio, with optimal results achieved around 0.6. Very low ratios might lead to overfitting, while very high ratios can result in suboptimal recovery due to excessive missing information.
  • Iteration Count: While performance improves with more iterations, significant gains plateau after three layers (L=3). The authors choose L=3 to balance performance and computational resources.
  • Future Work: The paper suggests that the MMR module's ability to generate arbitrary upsampling points could be further explored. Additionally, the current framework could be extended to handle more complex scenarios or different types of 3D data.

Summary updated

BibTeX
@inproceedings{nie2025spu,
  author = {Ziming Nie and Qiao Wu and Chenlei Lv and Siwen Quan and Zhaoshuai Qi and Muze Wang and Jiaqi Yang},
  title = {{SPU-IMR: Self-supervised Arbitrary-scale Point Cloud Upsampling via Iterative Mask-recovery Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32667},
}
Figure