Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis

Kang Han, Wei Xiang

Introduction

Neural Radiance Fields (NeRF) have revolutionized the field of 3D scene reconstruction and novel view synthesis by enabling high-fidelity rendering from sparse image sets. However, traditional NeRF models face significant hurdles in two areas: computational efficiency and the accurate representation of complex, view-dependent reflections. While recent "feature-based" methods attempt to accelerate training by storing data in grids or hash tables, they often struggle with spatial continuity or fail to capture the intricate physics of light transport on specular surfaces.

This paper introduces the Neural Radiance Feature Field (NRFF), a novel framework that combines a multiscale tensor decomposition for scene geometry with a physics-inspired "Rendering Equation Encoding" (REE). By explicitly modeling the specular integral in feature space and utilizing a hierarchical geometry representation, NRFF achieves state-of-the-art results, outperforming existing methods like TensoRF by over 1 dB in PSNR while maintaining high parameter efficiency.

Methodology

NRFF introduces two primary technical innovations designed to improve both the geometric representation and the appearance modeling of 3D scenes.

1. Multiscale Tensor Decomposition (MTD)

Existing tensor-based methods, such as TensoRF, typically operate at a single resolution, which can miss the benefits of coarse-to-fine learning. NRFF represents the scene using a hierarchy of 16 resolution scales. At each level, the 3D space is decomposed into 2D plane feature maps and 1D line vectors.

This multiscale approach allows the model to capture smooth, global structures at coarser scales while progressively refining intricate details at finer scales. Unlike hash-based methods that can suffer from spatial artifacts due to hash collisions, this tensor decomposition preserves local coherence, leading to cleaner and more stable geometry.

2. Rendering Equation Encoding (REE)

Standard NeRF models often treat view-dependent effects as a black-box learning problem, feeding camera directions into a Multi-Layer Perceptron (MLP). NRFF instead encodes the rendering equation itself into the feature space.

The method utilizes Anisotropic Spherical Gaussian (ASG) mixtures to model light interaction. The process involves:

  • Reparameterization: The view direction is reflected across the surface normal to align the encoding with physical specular reflection behavior.
  • Feature Integration: The model samples 128 "lobes" on a sphere to represent incoming light. This provides the MLP with a feature vector that approximates the specular integral, effectively "informing" the network of the physics of light transport before processing begins.

Experimental Evaluation

The model was evaluated on three standard benchmarks: NeRF Synthetic, NSVF Synthetic, and the real-world Tanks & Temples dataset.

Quantitative Results

NRFF demonstrated significant improvements in image quality across all benchmarks:

  • Synthetic Data: NRFF improved PSNR by over 1 dB on both NeRF and NSVF synthetic datasets compared to previous state-of-the-art methods.
  • Real-World Data: On the Tanks & Temples dataset, NRFF achieved a PSNR of 29.05, significantly outperforming the original NeRF (25.78) and TensoRF (28.56).

Qualitative Analysis

Visual results indicate that NRFF produces much sharper textures and more realistic specular highlights. This is particularly evident in scenes with complex materials, such as the metallic surfaces in the "Materials" scene or the dynamic reflections on water in the "Ship" scene.

Key Insights

  • Physics-Informed Learning: By providing the network with a feature vector that approximates the specular integral via ASGs, the learning task is simplified, allowing the MLP to focus on refining details rather than learning basic optical laws from scratch.
  • Parameter Efficiency: The multiscale approach is surprisingly efficient. A 16-level NRFF model with 8.5M parameters outperformed a single-scale TensoRF model with 13M parameters, suggesting that hierarchical representations are better at compressing scene information.
  • Convergence Speed: Coarse scales capture the general scene shape rapidly, allowing the model to reach high-quality results faster than single-scale alternatives.

Limitations and Future Work

While NRFF provides superior image quality, it introduces some computational trade-offs. The use of a larger MLP (9-10 layers) and the overhead of interpolating across 16 scales results in a rendering speed of approximately 2–3 seconds per frame. While faster than the original NeRF, it is slower than "instant" methods like Instant-NGP.

Future research directions include optimizing the feature lookup process through hardware-accelerated bilinear interpolation and custom CUDA kernels. Additionally, the authors suggest extending the REE framework to handle dynamic relighting under varying environment maps, further bridging the gap between neural rendering and traditional computer graphics.

Summary updated

Abstract

Rendering novel views from captured multi-view images has made considerable progress since the emergence of the neural radiance field. This paper aims to further advance the quality of view synthesis by proposing a novel approach dubbed the neural radiance feature field (NRFF). We first propose a multiscale tensor decomposition scheme to organize learnable features so as to represent scenes from coarse to fine scales. We demonstrate many benefits of the proposed multiscale representation, including more accurate scene shape and appearance reconstruction, and faster convergence compared with the single-scale representation. Instead of encoding view directions to model view-dependent effects, we further propose to encode the rendering equation in the feature space by employing the anisotropic spherical Gaussian mixture predicted from the proposed multiscale representation. The proposed NRFF improves state-of-the-art rendering results by over 1 dB in PSNR on both the NeRF and NSVF synthetic datasets. A significant improvement has also been observed on the real-world Tanks & Temples dataset. Code can be found at https://github.com/imkanghan/nrff.
BibTeX
@inproceedings{han2023multiscale,
  author = {Kang Han and Wei Xiang},
  title = {{Multiscale Tensor Decomposition and Rendering Equation Encoding for View Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure