PVT: An Implicit Surface Reconstruction Framework via Point Voxel Geometric-Aware Transformer

Chuanmao Fan, Chenxi Zhao, Ye DuanWACV 2025

TL;DR

This paper introduces PVT, a novel framework for 3D surface reconstruction that seamlessly fuses point and voxel representations using a geometry-aware transformer. By effectively integrating local geometric details from point clouds with global context from volumetric grids, PVT achieves state-of-the-art performance on various reconstruction tasks, outperforming existing methods.

What Problem Does This Paper Address?

3D surface reconstruction from unorganized point clouds is a crucial task with applications in robotics, VR, and animation. While deep learning methods have shown great promise, they often struggle to balance the representation of fine-grained local geometry with efficient global context encoding. Traditional voxel-based methods suffer from memory issues and discretization loss, while point-based methods can be less efficient for neighborhood queries. Existing hybrid approaches have explored fusing point and voxel data, but this paper identifies a need for a more seamless and geometrically aware fusion mechanism to achieve lossless representation and improved reconstruction quality.

Methods

The core of the PVT framework is a novel Point-Voxel Transformer (PVT) block. This block is designed to fuse point and voxel features bidirectionally using transformers. Key components include:

  • Geo-attention pooling: This mechanism fuses point features into voxel representations. It uses a "geo-attention" transformer that leverages spherical embeddings to encode the relative geometric relationship between points and voxel centers. This is a departure from simpler averaging methods.
  • Geo-attention interpolation: This is the inverse operation, fusing voxel features back to points. It also utilizes transformers and spherical embeddings.
  • Asymmetric U-Net Architecture: The PVT encoder employs a U-Net-like structure where the downsampling path is built with PVT blocks, capturing multi-scale volumetric features. The upsampling path uses standard 3D convolutions.
  • Implicit Field Decoder: The extracted multi-level features are fed into an MLP-based decoder that predicts either an occupancy probability or an unsigned distance field (UDF) for a given query point.
  • Spherical Embedding: This technique, detailed in Section 3.3, parameterizes local point positions within a voxel using spherical coordinates (radius, azimuth, elevation) relative to the voxel center. This embedding captures local shape geometry more effectively than simple positional encodings.
  • Random Fourier Positional Embedding: For the query points fed into the decoder, random Fourier features are used to help the network learn high-frequency shape details.

Datasets

The paper evaluates PVT on several standard datasets:

  • ShapeNet: Used for closed surface reconstruction, including both general classes and specific complex shapes like cars.
  • FAUST and Multi-Garment: Used for open surface reconstruction, particularly for human and garment data.
  • Gibson: Used to demonstrate the framework's generalization capabilities on large-scale indoor scenes.

Key Insights and Ideas

  • Geometry-aware fusion is crucial: The paper argues that simply averaging point features into voxels or using trilinear interpolation for voxel-to-point conversion leads to information loss. The proposed geo-attention mechanism with spherical embeddings allows for a more informed and geometrically sensitive fusion.
  • Spherical embeddings capture local geometry effectively: By encoding points' positions relative to voxel centers in spherical coordinates, the network gains a better understanding of local surface orientation and curvature, which is vital for accurate reconstruction.
  • Transformer's ability to model long-range dependencies is beneficial: While transformers are known for their success in NLP, their application here allows for capturing relationships between points and voxels across different scales and spatial locations, leading to more robust representations.
  • Hybrid approach offers the best of both worlds: PVT successfully combines the strengths of point-based methods (local detail) and voxel-based methods (global context and efficient neighborhood processing) without the significant drawbacks of each.

Key Results

  • State-of-the-art performance: PVT achieves superior results on ShapeNet for closed surface reconstruction, outperforming methods like GeoUDF, POCO, and GridFormer in terms of Chamfer distance and F-score, especially under noisy conditions.
  • Improved open surface reconstruction: The method also demonstrates strong performance on open surface reconstruction tasks using the Multi-Garment dataset.
  • Accurate reconstruction of complex shapes: PVT shows impressive results for shapes with intricate inner structures, such as cars, capturing finer details than baselines.
  • Generalization to large scenes: The framework demonstrates good generalization capabilities by successfully reconstructing large indoor scenes from the Gibson dataset.
  • Ablation studies highlight component importance: Experiments show that using spherical encoding (SE) and Fourier encoding (FE) significantly improves performance compared to a baseline PVT without these features.

Limitations and Future Work

  • Failure cases: The paper acknowledges a failure case in occupancy field reconstruction for a specific ShapeNet object, suggesting that performance can drop in certain scenarios.
  • Computational cost: While PVT aims for efficiency, transformer-based methods can still be computationally intensive. Further optimization might be explored.
  • Future directions: The authors suggest exploring the framework for other 3D tasks beyond surface reconstruction, such as semantic segmentation or object detection, and investigating more advanced transformer architectures for even better fusion capabilities.

Summary updated

BibTeX
@inproceedings{fan2025pvt,
  author = {Chuanmao Fan and Chenxi Zhao and Ye Duan},
  title = {{PVT: An Implicit Surface Reconstruction Framework via Point Voxel Geometric-Aware Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure