Point Cloud Color Upsampling with Attention-Based Coarse Colorization and Refinement

Kohei Matsuzaki, Keisuke NonakaWACV 2025

TL;DR

This paper introduces a novel attention-based method for point cloud color upsampling, addressing the challenge of generating high-resolution, realistic colors for sparse 3D data. By learning to weigh the importance of neighboring low-resolution points, their approach achieves significantly improved color quality compared to existing methods, making it valuable for applications like VR/AR and 3D reconstruction.

What Problem Does This Paper Address?

Point clouds, essential for 3D applications like virtual reality, augmented reality, and autonomous navigation, are often captured at low densities by sensors. While geometry upsampling (increasing point density) has been extensively studied, the problem of color upsampling – assigning accurate colors to these newly generated points – remains less explored. Existing methods often rely on simple geometric proximity or fixed weighting schemes, leading to blurry colors and artifacts, especially at object boundaries where color can change rapidly. This paper tackles this challenge by proposing a more sophisticated, learning-based approach to color upsampling that can handle complex textures and color transitions.

Methods

The proposed method, named "Attention-based Coarse Colorization and Refinement" (ACCR), consists of three main stages:

  1. Feature Extraction: A Point Transformer v3 (PTv3) network is used to extract rich, point-wise features from the low-resolution (LR) colored point cloud. This network leverages a U-Net-like architecture with attention mechanisms to capture spatial relationships.
  2. Attention-Based Coarse Colorization: For each high-resolution (HR) point, the method identifies its k-nearest neighbors in the LR point cloud. Crucially, instead of using fixed weights based on distance, a deep neural network learns to predict attention weights that signify the importance of each LR neighbor for coloring the HR point. These learned weights are then used to compute a weighted average of the LR point colors, producing a coarse color estimate for the HR point.
  3. Color Refinement: To further enhance the color accuracy, the method predicts color offsets. It aggregates features from the LR neighbors (using the same learned attention weights) and feeds them into a linear layer to predict these offsets. The final HR point color is obtained by adding these predicted offsets to the coarse color estimates. This two-stage approach (coarse prediction followed by refinement) is designed to be robust and accurate.

Datasets

The authors evaluate their method on two real-world datasets:

  • OmniObject3D: A dataset containing 6,000 objects across 190 categories, scanned in real-world environments.
  • ScanNet-v2: A dataset comprising 1,513 indoor scenes captured from over 707 unique real-world indoor environments.

These datasets are used to generate both high-resolution (HR) and low-resolution (LR) point clouds by applying voxelization with different grid sizes.

Key Insights and Ideas

  • Learned Attention for Color: The core innovation is the use of learned attention weights to determine the importance of neighboring LR points for color upsampling. This moves beyond simple distance-based weighting, allowing the model to adapt to complex color distributions and textures.
  • Coarse-to-Fine Strategy: The two-stage approach of coarse colorization followed by refinement is effective. The coarse stage provides a good initial estimate, and the refinement stage corrects for residual errors, leading to higher overall accuracy.
  • Robustness to Unstructured Data: Unlike some methods that require voxelized inputs, the proposed approach can directly handle unstructured point clouds, demonstrating its versatility.
  • Generalization: The method shows good generalization capabilities, performing well even when trained on one upsampling scale and tested on others.

Key Results

  • Quantitative Performance: The proposed method consistently outperforms state-of-the-art methods (NN, k-NN, WAAN, CU-Net, AttNet) in terms of Peak Signal-to-Noise Ratio (PSNR) on both OmniObject3D and ScanNet-v2 datasets across various upsampling rates (2x to 8x). For example, on OmniObject3D at 8x upsampling, the proposed method achieves a PSNR of 36.07 dB, significantly higher than AttNet's 33.80 dB.
  • Qualitative Results: Visualizations show that the proposed method produces sharper and more accurate colors, with fewer blurring artifacts and noise, especially near color boundaries, compared to existing methods.
  • Ablation Studies: Experiments confirm the effectiveness of both the attention-based coarse colorization and the feature aggregation for refinement, with each component contributing to improved performance.

Limitations and Future Work

  • Isolated Points: The method's reliance on neighboring points makes it challenging to accurately color isolated points where no nearby LR points exist.
  • Computational Complexity: The k-nearest neighbor search introduces significant computational cost.
  • Future Directions: The authors suggest exploring methods to incorporate more global information to improve performance for isolated points and investigating approximation techniques for the k-nearest neighbor search to reduce computational complexity.

Summary updated

BibTeX
@inproceedings{matsuzaki2025point,
  author = {Kohei Matsuzaki and Keisuke Nonaka},
  title = {{Point Cloud Color Upsampling with Attention-Based Coarse Colorization and Refinement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure