PET-NeuS: Positional Encoding Tri-Planes for Neural Surfaces

Yiqun Wang, Ivan Skorokhodov, Peter Wonka

Abstract

Neural surface reconstruction has seen significant progress with coordinate-based MLPs, yet capturing fine-grained geometric details remains a challenge. We present PET-NeuS, a framework that combines the memory efficiency of tri-plane representations with a novel modulated positional encoding (MPE) strategy. By employing self-attention to manage multi-frequency features, PET-NeuS effectively captures intricate details while suppressing the high-frequency noise typical of explicit grid-based structures.

1. Introduction

Reconstructing 3D surfaces from 2D images is a fundamental task in computer vision. While methods like NeuS utilize Coordinate-based Multi-Layer Perceptrons (MLPs) to represent Signed Distance Functions (SDFs), they often struggle with high-frequency details due to the inherent "smoothness bias" of MLPs. Conversely, explicit structures like tri-planes offer high expressiveness but are prone to noise and artifacts.

High-fidelity reconstruction of a complex architectural scene using PET-NeuS.

PET-NeuS bridges this gap by introducing a hybrid approach that maintains the detail of tri-planes while enforcing the necessary smoothness for clean surface reconstruction. It addresses the "smoothing paradox"—the need for global smoothness alongside local sharpness—by using self-attention to decide which neighboring features are relevant for smoothing, thereby preserving edges while eliminating noise.

2. Methodology

The PET-NeuS architecture introduces three primary innovations to the standard neural surface pipeline:

2.1 Tri-Plane Representation

The model utilizes a tri-plane architecture (XY, YZ, and XZ planes) to store features. This allows for efficient local information lookup, significantly increasing the model's capacity for detail compared to global MLPs.

2.2 Modulated Positional Encoding (MPE)

Standard positional encoding (PE) uses fixed sinusoidal functions. PET-NeuS introduces Modulated Positional Encoding, where tri-plane features are used to modulate (multiply) these sinusoids. This creates a learnable filter that integrates grid-based features with frequency information, suppressing noise while preserving signal.

2.3 Self-Attention Convolution (SAC)

To manage different levels of detail without blurring across object boundaries, the framework employs Self-Attention Convolution. Unlike standard Gaussian blurring, SAC uses varying window sizes and attention mechanisms to smooth features locally, respecting depth discontinuities and preserving sharp edges.

3. Experiments and Results

PET-NeuS was evaluated on the DTU, NeRF-Synthetic, and CO3D datasets, demonstrating superior performance in both synthetic and real-world "in-the-wild" captures.

Qualitative comparison of PET-NeuS against the NeuS baseline and various ablation components (PE, MPE).

3.1 Quantitative Performance

The method demonstrates a significant leap in accuracy. On the NeRF-Synthetic dataset, PET-NeuS improved the Chamfer distance (a measure of surface error) by 57% over the NeuS baseline (0.84 vs. 1.97). On the real-world DTU dataset, it achieved a 15.5% improvement in surface error.

3.2 Qualitative Analysis

As shown in the comparisons, PET-NeuS successfully reconstructs complex geometries—such as thin wires, tiny holes in statues, and sharp mechanical edges—that previous methods often fail to resolve or represent as noisy blobs.

3.3 Efficiency

Training PET-NeuS takes approximately 9 hours on a single NVIDIA V100. While not real-time, it is significantly faster than competitors like HF-NeuS, which can take up to 20 hours, proving that a shallow 3-layer MLP combined with a smart feature grid is both faster and more accurate.

4. Discussion and Conclusion

PET-NeuS demonstrates that treating tri-plane features as coefficients for positional encoding provides a robust theoretical and practical bridge between explicit and implicit representations. While the model is highly expressive, it can occasionally overfit to complex lighting, suggesting that future work should focus on further distinguishing between sensor noise and true geometric detail. Overall, PET-NeuS sets a new standard for high-fidelity, efficient neural surface reconstruction.

Summary updated

Abstract

A signed distance function (SDF) parametrized by an MLP is a common ingredient of neural surface reconstruction. We build on the successful recent method NeuS to extend it by three new components. The first component is to borrow the tri-plane representation from EG3D and represent signed distance fields as a mixture of tri-planes and MLPs instead of representing it with MLPs only. Using tri-planes leads to a more expressive data structure but will also introduce noise in the reconstructed surface. The second component is to use a new type of positional encoding with learnable weights to combat noise in the reconstruction process. We divide the features in the tri-plane into multiple frequency scales and modulate them with sin and cos functions of different frequencies. The third component is to use learnable convolution operations on the tri-plane features using self-attention convolution to produce features with different frequency bands. The experiments show that PET-NeuS achieves high-fidelity surface reconstruction on standard datasets. Following previous work and using the Chamfer metric as the most important way to measure surface reconstruction quality, we are able to improve upon the NeuS baseline by 57% on Nerf-synthetic (0.84 compared to 1.97) and by 15.5% on DTU (0.71 compared to 0.84). The qualitative evaluation reveals how our method can better control the interference of high-frequency noise. Code available at \url{https://github.com/yiqun-wang/PET-NeuS}.
BibTeX
@inproceedings{wang2023pet,
  author = {Yiqun Wang and Ivan Skorokhodov and Peter Wonka},
  title = {{PET-NeuS: Positional Encoding Tri-Planes for Neural Surfaces}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01212},
}
Figure