Abstract
Neural surface reconstruction has seen significant progress with coordinate-based MLPs, yet capturing fine-grained geometric details remains a challenge. We present PET-NeuS, a framework that combines the memory efficiency of tri-plane representations with a novel modulated positional encoding (MPE) strategy. By employing self-attention to manage multi-frequency features, PET-NeuS effectively captures intricate details while suppressing the high-frequency noise typical of explicit grid-based structures.
1. Introduction
Reconstructing 3D surfaces from 2D images is a fundamental task in computer vision. While methods like NeuS utilize Coordinate-based Multi-Layer Perceptrons (MLPs) to represent Signed Distance Functions (SDFs), they often struggle with high-frequency details due to the inherent "smoothness bias" of MLPs. Conversely, explicit structures like tri-planes offer high expressiveness but are prone to noise and artifacts.
PET-NeuS bridges this gap by introducing a hybrid approach that maintains the detail of tri-planes while enforcing the necessary smoothness for clean surface reconstruction. It addresses the "smoothing paradox"—the need for global smoothness alongside local sharpness—by using self-attention to decide which neighboring features are relevant for smoothing, thereby preserving edges while eliminating noise.
2. Methodology
The PET-NeuS architecture introduces three primary innovations to the standard neural surface pipeline:
2.1 Tri-Plane Representation
The model utilizes a tri-plane architecture (XY, YZ, and XZ planes) to store features. This allows for efficient local information lookup, significantly increasing the model's capacity for detail compared to global MLPs.
2.2 Modulated Positional Encoding (MPE)
Standard positional encoding (PE) uses fixed sinusoidal functions. PET-NeuS introduces Modulated Positional Encoding, where tri-plane features are used to modulate (multiply) these sinusoids. This creates a learnable filter that integrates grid-based features with frequency information, suppressing noise while preserving signal.
2.3 Self-Attention Convolution (SAC)
To manage different levels of detail without blurring across object boundaries, the framework employs Self-Attention Convolution. Unlike standard Gaussian blurring, SAC uses varying window sizes and attention mechanisms to smooth features locally, respecting depth discontinuities and preserving sharp edges.
3. Experiments and Results
PET-NeuS was evaluated on the DTU, NeRF-Synthetic, and CO3D datasets, demonstrating superior performance in both synthetic and real-world "in-the-wild" captures.
3.1 Quantitative Performance
The method demonstrates a significant leap in accuracy. On the NeRF-Synthetic dataset, PET-NeuS improved the Chamfer distance (a measure of surface error) by 57% over the NeuS baseline (0.84 vs. 1.97). On the real-world DTU dataset, it achieved a 15.5% improvement in surface error.
3.2 Qualitative Analysis
As shown in the comparisons, PET-NeuS successfully reconstructs complex geometries—such as thin wires, tiny holes in statues, and sharp mechanical edges—that previous methods often fail to resolve or represent as noisy blobs.
3.3 Efficiency
Training PET-NeuS takes approximately 9 hours on a single NVIDIA V100. While not real-time, it is significantly faster than competitors like HF-NeuS, which can take up to 20 hours, proving that a shallow 3-layer MLP combined with a smart feature grid is both faster and more accurate.
4. Discussion and Conclusion
PET-NeuS demonstrates that treating tri-plane features as coefficients for positional encoding provides a robust theoretical and practical bridge between explicit and implicit representations. While the model is highly expressive, it can occasionally overfit to complex lighting, suggesting that future work should focus on further distinguishing between sensor noise and true geometric detail. Overall, PET-NeuS sets a new standard for high-fidelity, efficient neural surface reconstruction.
Summary updated