TL;DR
This paper introduces PET-NeuS, a novel method for reconstructing high-fidelity 3D surfaces from images. By combining a tri-plane representation with specialized positional encoding and self-attention convolutions, PET-NeuS significantly improves surface detail and reduces noise compared to existing methods, achieving state-of-the-art results on standard benchmarks.
What Problem Does This Paper Address?
Reconstructing detailed and accurate 3D surfaces from 2D images is a long-standing challenge in computer vision. While methods like NeRF and NeuS have made significant strides using neural networks to represent scenes as implicit functions (like Signed Distance Functions or SDFs), they often struggle to capture fine-grained geometric details and are susceptible to noise. Existing approaches that use more expressive data structures like tri-planes (inspired by EG3D) can improve detail but often introduce unwanted noise due to the discrete nature of these representations. The core problem is balancing the expressiveness needed for fine details with the robustness required to handle noise and ensure smooth, accurate surface reconstruction.
Methods
The PET-NeuS framework builds upon the NeuS method and introduces three key innovations:
- Tri-Plane Representation for SDFs: Instead of relying solely on MLPs, PET-NeuS uses a tri-plane representation (three 2D feature planes) to encode the SDF. This allows for a more expressive representation of local geometry, capturing finer details. However, this also introduces challenges with noise.
- Novel Positional Encoding: To combat the noise introduced by tri-planes, the authors propose a new positional encoding strategy. This method modulates the tri-plane features using sinusoidal functions at different frequencies. This effectively injects frequency information into the grid-based representation, helping to suppress high-frequency noise and improve surface smoothness.
- Self-Attention Convolution for Multi-Frequency Features: To further enhance feature learning and noise reduction, PET-NeuS employs self-attention convolution (SAC) with varying window sizes. This allows the network to generate features across different frequency bands, which are then combined with the modulated tri-plane features. This approach is more effective than simple Gaussian smoothing, which can blur features across depth discontinuities.
The overall framework integrates these components, using the modulated and multi-frequency tri-plane features as input to an MLP that predicts the final SDF.
Datasets
The authors evaluate PET-NeuS on two standard datasets:
- NeRF-Synthetic Dataset: This dataset provides posed multi-view images of scenes with detailed and sharp features, suitable for evaluating the reconstruction of fine geometry.
- DTU Dataset: This dataset consists of real-world multi-view images and is known for its non-Lambertian surfaces, making it a good testbed for evaluating a method's robustness to noise.
Key Insights and Ideas
- Tri-planes are expressive but noisy: While tri-planes offer a more compact and expressive way to represent geometry than pure MLPs, their discrete nature inherently introduces noise. This highlights a fundamental trade-off in 3D reconstruction.
- Frequency-aware encoding is crucial for grid-based representations: Directly applying standard positional encoding to tri-planes isn't optimal. The proposed method of modulating tri-plane features with learned sinusoidal functions at different frequencies is key to suppressing noise and improving fidelity.
- Self-attention convolution effectively captures multi-frequency information: Using SAC with different window sizes allows the network to learn features at various scales, which is more robust and effective than simple multi-resolution tri-planes for handling complex scene structures and noise.
- Geometric initialization matters: The paper emphasizes that proper initialization of the tri-plane representation is critical for convergence and avoiding artifacts, especially compared to random initialization or standard MLP initialization.
Key Results
- Quantitative Improvements: PET-NeuS significantly outperforms previous state-of-the-art methods like NeuS and HF-NeuS on both the NeRF-Synthetic and DTU datasets.
- On NeRF-Synthetic, PET-NeuS improves upon the NeuS baseline by 25% (Chamfer distance of 0.84 vs. 1.12).
- On DTU, it improves by 14% (Chamfer distance of 0.75 vs. 0.87).
- Qualitative Improvements: The method produces visually superior reconstructions with finer details and less noise, as demonstrated in figures showing reconstructions of objects like Lego models, chairs, and Buddhas. The paper highlights better control over high-frequency noise interference.
- Efficiency: PET-NeuS achieves faster training times (9 hours on DTU) compared to some baselines like NeuS (16 hours) and HF-NeuS (20 hours).
Limitations and Future Work
- Computation Time: While faster than some competitors, the authors acknowledge that computation time is still a limitation. They suggest that improving computation time by one or two orders of magnitude without sacrificing quality would be an exciting future direction.
- Detail vs. Noise Trade-off: The paper notes a persistent trade-off between reconstructing fine details and introducing high-frequency noise in flatter areas. More expressive architectures can sometimes lead to overfitting and hallucinating details. Investigating this trade-off from a theoretical perspective is suggested.
- Societal Impact: The authors state they do not expect a noteworthy negative societal impact from this research.
Summary updated