PET-NeuS: Positional Encoding Triplanes for Neural Surfaces

Yiqun Wang, Ivan Skorokhodov, Peter Wonka

TL;DR

This paper introduces PET-NeuS, a novel method for reconstructing high-fidelity 3D surfaces from images. By combining a tri-plane representation with specialized positional encoding and self-attention convolutions, PET-NeuS significantly improves surface detail and reduces noise compared to existing methods, achieving state-of-the-art results on standard benchmarks.

What Problem Does This Paper Address?

Reconstructing detailed and accurate 3D surfaces from 2D images is a long-standing challenge in computer vision. While methods like NeRF and NeuS have made significant strides using neural networks to represent scenes as implicit functions (like Signed Distance Functions or SDFs), they often struggle to capture fine-grained geometric details and are susceptible to noise. Existing approaches that use more expressive data structures like tri-planes (inspired by EG3D) can improve detail but often introduce unwanted noise due to the discrete nature of these representations. The core problem is balancing the expressiveness needed for fine details with the robustness required to handle noise and ensure smooth, accurate surface reconstruction.

Methods

The PET-NeuS framework builds upon the NeuS method and introduces three key innovations:

  1. Tri-Plane Representation for SDFs: Instead of relying solely on MLPs, PET-NeuS uses a tri-plane representation (three 2D feature planes) to encode the SDF. This allows for a more expressive representation of local geometry, capturing finer details. However, this also introduces challenges with noise.
  2. Novel Positional Encoding: To combat the noise introduced by tri-planes, the authors propose a new positional encoding strategy. This method modulates the tri-plane features using sinusoidal functions at different frequencies. This effectively injects frequency information into the grid-based representation, helping to suppress high-frequency noise and improve surface smoothness.
  3. Self-Attention Convolution for Multi-Frequency Features: To further enhance feature learning and noise reduction, PET-NeuS employs self-attention convolution (SAC) with varying window sizes. This allows the network to generate features across different frequency bands, which are then combined with the modulated tri-plane features. This approach is more effective than simple Gaussian smoothing, which can blur features across depth discontinuities.

The overall framework integrates these components, using the modulated and multi-frequency tri-plane features as input to an MLP that predicts the final SDF.

Datasets

The authors evaluate PET-NeuS on two standard datasets:

  • NeRF-Synthetic Dataset: This dataset provides posed multi-view images of scenes with detailed and sharp features, suitable for evaluating the reconstruction of fine geometry.
  • DTU Dataset: This dataset consists of real-world multi-view images and is known for its non-Lambertian surfaces, making it a good testbed for evaluating a method's robustness to noise.

Key Insights and Ideas

  • Tri-planes are expressive but noisy: While tri-planes offer a more compact and expressive way to represent geometry than pure MLPs, their discrete nature inherently introduces noise. This highlights a fundamental trade-off in 3D reconstruction.
  • Frequency-aware encoding is crucial for grid-based representations: Directly applying standard positional encoding to tri-planes isn't optimal. The proposed method of modulating tri-plane features with learned sinusoidal functions at different frequencies is key to suppressing noise and improving fidelity.
  • Self-attention convolution effectively captures multi-frequency information: Using SAC with different window sizes allows the network to learn features at various scales, which is more robust and effective than simple multi-resolution tri-planes for handling complex scene structures and noise.
  • Geometric initialization matters: The paper emphasizes that proper initialization of the tri-plane representation is critical for convergence and avoiding artifacts, especially compared to random initialization or standard MLP initialization.

Key Results

  • Quantitative Improvements: PET-NeuS significantly outperforms previous state-of-the-art methods like NeuS and HF-NeuS on both the NeRF-Synthetic and DTU datasets.
  • On NeRF-Synthetic, PET-NeuS improves upon the NeuS baseline by 25% (Chamfer distance of 0.84 vs. 1.12).
  • On DTU, it improves by 14% (Chamfer distance of 0.75 vs. 0.87).
  • Qualitative Improvements: The method produces visually superior reconstructions with finer details and less noise, as demonstrated in figures showing reconstructions of objects like Lego models, chairs, and Buddhas. The paper highlights better control over high-frequency noise interference.
  • Efficiency: PET-NeuS achieves faster training times (9 hours on DTU) compared to some baselines like NeuS (16 hours) and HF-NeuS (20 hours).

Limitations and Future Work

  • Computation Time: While faster than some competitors, the authors acknowledge that computation time is still a limitation. They suggest that improving computation time by one or two orders of magnitude without sacrificing quality would be an exciting future direction.
  • Detail vs. Noise Trade-off: The paper notes a persistent trade-off between reconstructing fine details and introducing high-frequency noise in flatter areas. More expressive architectures can sometimes lead to overfitting and hallucinating details. Investigating this trade-off from a theoretical perspective is suggested.
  • Societal Impact: The authors state they do not expect a noteworthy negative societal impact from this research.

Summary updated

Abstract

The signed distance function (SDF) represented by an MLP network is commonly used for multi-view neural surface reconstruction. We build on the successful recent method NeuS to extend it by three new components. The first component is to borrow the Tri-plane representation from EG3D and represent signed distance fields as a mixture of tri-planes and MLPs instead of representing it with MLPs only. Discretizing the scene space with Tri-planes leads to a more expressive data structure but involving tri-planes will introduce noise due to discrete discontinuities. The second component is to use a new type of positional encoding with learnable weights to combat noise in the reconstruction process. We divide the features in the tri-plane into multiple frequency bands and modulate them with sin and cos functions of different frequency. The third component is to use learnable convolution operations on the tri-plane features using self-attention convolution to produce features with different frequency. The experiments show that PET-NeuS achieves high-fidelity surface reconstruction on standard datasets. Following previous work and using the Chamfer metric as the most important way to measure surface reconstruction quality, we are able to improve upon the NeuS baseline by 25\% on Nerf-synthetic (0.84 compared to 1.12) and by 14\% on DTU (0.75 compared to 0.87). The qualitative evaluation reveals how our method can better control the interference of high-frequency noise.
BibTeX
@inproceedings{wang2023pet,
  author = {Yiqun Wang and Ivan Skorokhodov and Peter Wonka},
  title = {{PET-NeuS: Positional Encoding Triplanes for Neural Surfaces}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01212},
}
Figure