PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces Using Permutohedral Lattices

Radu Alexandru Rosu, Sven Behnke

Abstract

Neural radiance-density field methods have become increasingly popular for the task of novel-view rendering. Their recent extension to hash-based positional encoding ensures fast training and inference with visually pleasing results. However, density-based methods struggle with recovering accurate surface geometry. Hybrid methods alleviate this issue by optimizing the density based on an underlying Signed Distance Function (SDF). However, current SDF methods are overly smooth and miss fine geometric details. In this work, we combine the strengths of these two lines of work in a novel hash-based implicit surface representation. We propose improvements to the two areas by replacing the voxel hash encoding with a permutohedral lattice which optimizes faster, especially for higher dimensions. We additionally propose a regularization scheme which is crucial for recovering high-frequency geometric detail. We evaluate our method on multiple datasets and show that we can recover geometric detail at the level of pores and wrinkles while using only RGB images for supervision. Furthermore, using sphere tracing we can render novel views at 30 fps on an RTX 3090.

Introduction

While Neural Radiance Fields (NeRFs) have revolutionized novel-view synthesis, they often struggle with "geometry." Standard density-based NeRFs frequently produce "cloudy" or noisy surfaces, especially in untextured or highly specular regions. Existing solutions generally fall into two camps: SDF-based methods (e.g., NeuS, VolSDF), which produce clean geometry but are slow and over-smooth, and hash-based methods (e.g., Instant-NGP), which are fast but suffer from noisy geometry.

PermutoSDF bridges this gap. It is a novel framework that achieves high-fidelity 3D reconstruction from multi-view images in just 30 minutes. By combining the efficiency of permutohedral lattice hashing with an SDF representation and a clever Lipschitz regularization scheme, it recovers fine geometric details like skin pores and wrinkles while enabling real-time rendering at 30 FPS.

Methodology

The core of PermutoSDF lies in three technical innovations designed to balance speed and geometric accuracy.

Permutohedral Lattice Hashing

Instead of the standard cubical voxel grids used in Instant-NGP, PermutoSDF utilizes a permutohedral lattice. In a $d$-dimensional space, a cube has $2^d$ vertices, but a simplex (the basic unit of a permutohedral lattice) has only $d+1$ vertices. This linear scaling makes the lattice significantly more efficient for 3D and 4D (spatio-temporal) tasks, reducing memory accesses and speeding up interpolation. As dimensionality increases, the performance gap between permutohedral lattices and voxel grids widens, favoring the lattice for complex scenes.

The PermutoSDF pipeline: (1) Ray sampling, (2) Slicing through the permutohedral lattice, (3) Feature extraction, (4) MLP processing, and (5) Final SDF and Color Field output.

Lipschitz Color Regularization

A key challenge in neural reconstruction is preventing the model from "cheating" by painting high-frequency details (like a wrinkle) onto a flat surface via the color network. To solve this, the authors softly constrain the color network to be smooth (Lipschitz continuous). Because the color network is forced to be smooth, the optimization is forced to place high-frequency details into the SDF geometry instead to minimize reconstruction loss.

Hybrid Regularization and Sphere Tracing

The training process is split into phases, beginning with a curvature loss to ensure smooth surfaces in ambiguous areas, followed by Lipschitz color regularization to extract fine details. Because the model learns an explicit SDF, the system can utilize sphere tracing for rendering. This is mathematically more efficient than the volumetric ray marching used in standard NeRFs, allowing for real-time performance.

Key Results

PermutoSDF was evaluated on challenging benchmarks including the DTU dataset, the Multiface dataset, and high-resolution synthetic head models.

PermutoSDF achieves high-fidelity reconstruction in 30 minutes, recovering fine details like wrinkles and skin texture from multi-view RGB images.
  • Geometric Fidelity: The method recovers details at the level of individual skin pores and fine wrinkles using only RGB images for supervision—details that baselines like NeuS and COLMAP often smooth over.
  • Quantitative Performance: On the DTU dataset, PermutoSDF achieved the lowest mean Chamfer distance (0.61 with masks, 0.68 without), outperforming NeuS, Instant-NGP, and COLMAP.
  • Efficiency: The system trains in approximately 30 minutes and renders at 30 FPS (1080p) on a consumer-grade RTX 3090.
  • Spatio-temporal 4D: The lattice's efficiency allows for the reconstruction of evolving surfaces (e.g., a running horse) by treating time as a fourth dimension, benefiting from the $d+1$ vertex scaling.

Limitations and Future Work

Despite its strengths, PermutoSDF faces challenges with mirror-like surfaces where the model may introduce geometric noise to explain complex reflections. Extremely thin, disconnected structures like individual strands of hair remain difficult to reconstruct perfectly. Future work includes extending occupancy grids to true 4D structures for further acceleration and implementing additional temporal consistency losses to improve the smoothness of 4D reconstructions over time.

Summary updated

Abstract

Neural radiance-density field methods have become increasingly popular for the task of novel-view rendering. Their recent extension to hash-based positional encoding ensures fast training and inference with visually pleasing results. However, density-based methods struggle with recovering accurate surface geometry. Hybrid methods alleviate this issue by optimizing the density based on an underlying SDF. However, current SDF methods are overly smooth and miss fine geometric details. In this work, we combine the strengths of these two lines of work in a novel hash-based implicit surface representation. We propose improvements to the two areas by replacing the voxel hash encoding with a permutohedral lattice which optimizes faster, especially for higher dimensions. We additionally propose a regularization scheme which is crucial for recovering high-frequency geometric detail. We evaluate our method on multiple datasets and show that we can recover geometric detail at the level of pores and wrinkles while using only RGB images for supervision. Furthermore, using sphere tracing we can render novel views at 30 fps on an RTX 3090. Code is publicly available at https://radualexandru.github.io/permuto_sdf.
BibTeX
@inproceedings{rosu2023permutosdf,
  author = {Radu Alexandru Rosu and Sven Behnke},
  title = {{PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces Using Permutohedral Lattices}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=uixscdXlgj},
}
Figure