Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling

Wanquan Feng, Jin Li, Hongrui Cai, Xiaonan Luo, Juyong Zhang

Abstract

In this paper, we propose Neural Points, a novel point cloud representation and apply it to the arbitrary-factored upsampling task. Different from traditional point cloud representation where each point only represents a position or a local plane in the 3D space, each point in Neural Points represents a local continuous geometric shape via neural fields. Therefore, Neural Points contain more shape information and thus have a stronger representation ability. Neural Points is trained with surface containing rich geometric details, such that the trained model has enough expression ability for various shapes. Specifically, we extract deep local features on the points and construct neural fields through the local isomorphism between the 2D parametric domain and the 3D local patch. In the final, local neural fields are integrated together to form the global surface. Experimental results show that Neural Points has powerful representation ability and demonstrate excellent robustness and generalization ability. With Neural Points, we can resample point cloud with arbitrary resolutions, and it outperforms the state-of-the-art point cloud upsampling methods.

Introduction

Point clouds are a fundamental data structure for 3D computer vision, yet their discrete nature poses significant challenges. Standard representations treat points as isolated $(x, y, z)$ coordinates, which leads to a loss of surface continuity when data is sparse or when zooming into fine details. Traditional upsampling methods—the process of increasing point density—often operate in a discrete-to-discrete fashion, producing a fixed number of output points (e.g., $4\times$ or $16\times$ the input). These methods frequently struggle with complex geometries, noise, and the inability to scale to arbitrary resolutions.

Neural Points introduces a paradigm shift by representing each point as the center of a local, continuous surface patch defined by a neural field. This approach enables "arbitrary-factored upsampling," allowing for the generation of high-resolution, smooth surfaces at any scale from sparse input data.

The Neural Points Representation

The core innovation of this work is the transition from static coordinates to dynamic local features that parameterize neural fields. Instead of a point being a simple dot, it becomes a "Neural Point" that encodes the local geometry of its neighborhood.

Overview of the Neural Points pipeline, from input point cloud to arbitrary-factored resampling via local neural fields.

Methodology

The construction of Neural Points involves several key technical steps:

  1. Local Isomorphism: Leveraging the topological property that small 3D manifold patches are isomorphic to 2D disks, the authors establish a bijective mapping between a 2D parametric domain and the 3D local patch.
  2. Neural Field Network: A shared Multi-Layer Perceptron (MLP) represents this mapping. It takes a 2D coordinate and a local feature vector as input to predict the corresponding 3D position on the surface.
  3. Feature Extraction: A Dynamic Graph CNN (DGCNN) backbone is utilized to extract deep local features, ensuring that each Neural Point captures essential geometric information like curvature and edges.
  4. Global Integration: To handle overlapping patches, a distance-based weighting scheme "stitches" local neural fields into a globally consistent continuous surface.
  5. Projection-based Loss: Training is guided by a projection strategy that pulls points toward the underlying continuous surface, avoiding the noise sensitivity of standard metrics like Chamfer distance.

Experimental Results

The authors evaluated Neural Points on diverse datasets, including high-detail models from Sketchfab, the PU-GAN dataset, and real-world data from iPhone depth sensors and LiDAR.

Performance and Visual Quality

Neural Points achieves state-of-the-art (SOTA) performance across standard metrics, including Chamfer Distance (CD), Hausdorff Distance (HD), and Point-to-Surface (P2F) distance. Unlike previous methods that often produce "clumpy" or noisy distributions at high upsampling factors, Neural Points maintains uniform and smooth surface reconstructions.

Qualitative and quantitative comparison of Neural Points against state-of-the-art upsampling methods on complex geometries.

The model demonstrates remarkable robustness to noise and excellent generalization to real-world data. Furthermore, its resolution-independent nature allows for extreme upsampling (up to $256\times$), reconstructing coherent shapes from inputs that are nearly unrecognizable.

Conclusion

Neural Points provides a powerful and memory-efficient representation for 3D point clouds. By treating points as local neural fields, it overcomes the limitations of discrete representations and fixed-factor upsampling. While currently focused on geometry, the entire trained model is only about 2.53 MB, making it highly practical. Future work aims to incorporate multi-modal data like color and global semantic context to further enhance its expressive power.

Summary updated

Abstract

In this paper, we propose Neural Points, a novel point cloud representation and apply it to the arbitrary-factored upsampling task. Different from traditional point cloud representation where each point only represents a position or a local plane in the 3D space, each point in Neural Points represents a local continuous geometric shape via neural fields. Therefore, Neural Points contain more shape information and thus have a stronger representation ability. Neural Points is trained with surface containing rich geometric details, such that the trained model has enough expression ability for various shapes. Specifically, we extract deep local features on the points and construct neural fields through the local isomorphism between the 2D parametric domain and the 3D local patch. In the final, local neural fields are integrated together to form the global surface. Experimental results show that Neural Points has powerful representation ability and demonstrate excellent robustness and generalization ability. With Neural Points, we can resample point cloud with arbitrary resolutions, and it outperforms the state-of-the-art point cloud upsampling methods. Code is available at https://github.com/WanquanF/NeuralPoints.
BibTeX
@inproceedings{feng2022neural,
  author = {Wanquan Feng and Jin Li and Hongrui Cai and Xiaonan Luo and Juyong Zhang},
  title = {{Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=2bkUd4-CBlu},
}
Figure