Anisotropic Neural Representation Learning for High-Quality Neural Rendering

Yifan Wang, Jun Xu, Yi Zeng, Yuan GongarXiv 2023 · DOI · arXiv

Abstract

Standard Neural Radiance Fields (NeRFs) typically represent 3D scenes using isotropic point features, assuming that volume density and latent features are independent of the viewing direction. However, this assumption fails to account for the directional nature of sampling intervals along camera rays, leading to "shape-radiance ambiguity," blurring, and aliasing. We propose a "plug-and-play" anisotropic neural representation that models density and features as view-dependent functions using Spherical Harmonics (SH). By factorizing representations into view-independent and view-dependent components and applying a novel anisotropy regularization loss, our method significantly improves rendering quality across various state-of-the-art NeRF architectures.

Introduction: The Problem of Directional Ambiguity

Standard NeRF models assign a single volume density ($\sigma$) and latent feature ($e$) to each point in space. These values are isotropic, meaning they remain constant regardless of the observer's perspective. In reality, a "point" sample in volume rendering represents a small interval along a ray. In complex scenes—such as those with fine textures, semi-transparent surfaces, or intricate geometry—the effective density of this interval can vary significantly depending on the viewing angle.

Forcing an isotropic representation on these directional intervals creates a fundamental ambiguity: the model cannot distinguish whether a visual effect is due to geometry (density) or appearance (color). This often results in "fuzzy" surfaces, blurry textures, and "floaters" in the reconstructed scene.

Conceptual comparison of isotropic vs. anisotropic representations (left) and qualitative rendering results on the Ship scene (right). The anisotropic approach recovers sharp details in ropes and textures that isotropic models miss.

Proposed Method: Anisotropic Neural Representation

To address these limitations, we introduce an anisotropic representation where geometry itself is view-dependent.

SH-Guided Features

Instead of predicting a single scalar for density, our primary MLP predicts coefficients ($k$ and $W$) for a set of Spherical Harmonics (SH). These basis functions allow us to represent signals on a sphere efficiently. For a given ray with direction $d$, the model queries these functions to obtain a view-dependent density $\sigma(d, k)$ and latent feature $e(d, W)$. This allows the representation to capture the "internal structure" of a sampling interval.

Anisotropy Regularization

Allowing geometry to be view-dependent introduces a risk of overfitting, where the model might create "fake" geometry to match training images from specific angles. To ensure geometric consistency, we introduce a point-wise regularization loss, $\mathcal{L}_{aniso}$. This loss penalizes the energy of higher-order SH components, encouraging the model to maintain an isotropic representation unless the training data provides strong evidence for anisotropy.

The proposed anisotropic neural representation pipeline. The first MLP ($\mathcal{F}_\theta$) predicts SH coefficients $k$ and $W$, which are combined with the viewing direction $d$ to produce anisotropic density and features.

Architecture Agnostic Integration

Our module is designed to be "plug-and-play." We demonstrate its versatility by integrating it into four prominent NeRF frameworks: Nerfacc, K-Planes, Tri-Mip, and Zip-NeRF.

Experimental Evaluation

We evaluated our method on two benchmark datasets:

  • Blender Dataset: 8 synthetic scenes used for high-detail reconstruction testing.
  • Mip-360 Dataset: 7 real-world unbounded scenes with complex backgrounds and intricate objects.

Results

The addition of the anisotropic module consistently improved performance across all baselines. On the Blender dataset, integrating the module into Nerfacc increased the PSNR from 33.06 to 34.08. On the real-world Mip-360 dataset, Zip-NeRF saw an improvement from 28.67 to 29.14 PSNR.

Qualitatively, the method excels at capturing fine details, such as the sharp edges of foliage, realistic specular reflections on metallic surfaces, and clear textures on distant objects. The regularization effectively reduces "shape-radiance ambiguity," leading to cleaner depth maps and fewer geometric artifacts.

Conclusion and Future Work

By moving beyond the isotropic assumption, our anisotropic neural representation provides a more accurate model for volume rendering in complex scenes. While the use of Spherical Harmonics introduces a minor computational overhead, the gains in rendering quality and geometric fidelity are substantial. Future work will explore alternative basis functions and optimization techniques for real-time applications on mobile hardware.

Summary updated

BibTeX
@misc{wang2023anisotropic,
  author = {Yifan Wang and Jun Xu and Yi Zeng and Yuan Gong},
  title = {{Anisotropic Neural Representation Learning for High-Quality Neural Rendering}},
  year = {2023},
  eprint = {2311.18311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18311},
}
Figure