Anisotropic Fourier Features for Neural Image-Based Rendering and Relighting

Huangjie Yu, Anpei Chen, Xin Chen, Lan Xu, Ziyu Shao, Jingyi Yu

Abstract

Coordinate-based Multilayer Perceptrons (MLPs) have become a cornerstone of neural rendering, yet they often struggle to capture high-frequency details due to spectral bias. While Fourier features can mitigate this, standard implementations are isotropic, treating all input dimensions equally. This paper introduces Anisotropic Fourier Features Mapping (AFFM), which assigns distinct bandwidths to different dimensions, such as spatial and angular coordinates. By leveraging an efficient Golden-Section Search (GSS) for bandwidth optimization, AFFM achieves significantly sharper rendering and relighting results. Our approach demonstrates a 7x speedup in parameter search and substantial gains in PSNR across various synthetic and real-world datasets.

Introduction

Modern neural rendering techniques rely on MLPs to map coordinates to scene properties. However, these networks naturally prefer learning smooth, low-frequency functions—a phenomenon known as "spectral bias." To address this, researchers employ Fourier features to project inputs into higher-dimensional spaces. Existing methods typically use a single bandwidth parameter for all dimensions. This isotropic approach is suboptimal for data like light fields, where spatial dimensions (textures) often contain much higher frequencies than angular dimensions (reflections).

Methodology

The core contribution of this work is the transition from isotropic to anisotropic mapping.

Anisotropic Fourier Features Mapping (AFFM)

AFFM replaces the scalar bandwidth parameter with a vector $\alpha_i$, allowing the network to adapt to the varying frequency characteristics of different input domains. This flexibility prevents the "compromise" seen in isotropic models, where one must choose between blurry textures or noisy reflections.

Theoretical analysis of Mean Integrated Squared Error (MISE) relative to bandwidth, data density, and signal complexity.

As shown in the theoretical analysis above, the optimal bandwidth is a function of both data density and signal complexity. AFFM allows the model to find the "sweet spot" for each dimension independently, minimizing the Mean Integrated Squared Error (MISE).

Finding optimal bandwidths manually is computationally expensive. We adapt the Golden-Section Search (GSS) to find these parameters in logarithmic time. Furthermore, we introduce an early-pruning heuristic: since better bandwidth candidates typically exhibit lower loss early in training, we can discard underperforming candidates after a few iterations, resulting in a 7x overall speedup compared to traditional grid search.

Results and Discussion

We evaluated AFFM on surface light field rendering and image-based relighting tasks using both synthetic and real-world datasets, including the "Lego" scene and the "Kneeling Knight" dataset.

Visual comparison of reconstruction quality between AFFM and Isotropic Fourier Feature Mapping (IFFM) on a high-frequency signal.

In comparative tests, AFFM consistently outperformed isotropic baselines. For instance, on the "Lego" scene, AFFM achieved a PSNR of 31.23, a significant improvement over the 25.10 reached by isotropic mapping. Visually, AFFM preserves intricate textures and sharp specular highlights that are typically lost or blurred by other methods. In relighting tasks, AFFM improved PSNR by nearly 3 dB on the Knight dataset, demonstrating its robustness in handling complex light transport.

Conclusion

Anisotropic Fourier Features Mapping provides a principled way to handle the heterogeneous frequency distributions found in multi-dimensional rendering data. By decoupling bandwidths and optimizing them efficiently via GSS, AFFM enables high-fidelity neural rendering with reduced computational overhead. Future work includes automating the grouping of dimensions and exploring the integration of anisotropic features into volumetric representations like NeRF.

Summary updated

Abstract

Recent neural rendering techniques have greatly benefited image-based modeling and relighting tasks. They provide a continuous, compact, and parallelable representation by modeling the plenoptic function as multilayer perceptrons (MLPs). However, vanilla MLPs suffer from spectral biases on multidimensional datasets. Recent rescues based on isotropic Fourier features mapping mitigate the problem but still fall short of handling heterogeneity across different dimensions, causing imbalanced regression and visual artifacts such as excessive blurs. We present an anisotropic random Fourier features (RFF) mapping scheme to tackle spectral biases. We first analyze the influence of bandwidth from a different perspective: we show that the optimal bandwidth exhibits strong correlations with the frequency spectrum of the training data across various dimensions. We then introduce an anisotropic feature mapping scheme with multiple bandwidths to model the multidimensional signal characteristics. We further propose an efficient bandwidth searching scheme through iterative golden-section search that can significantly reduce the training overload from polynomial time to logarithm. Our anisotropic scheme directly applies to neural surface light-field rendering and image-based relighting. Comprehensive experiments show that our scheme can more faithfully model lighting conditions and object features as well as preserve fine texture details and smooth view transitions even when angular and spatial samples are highly imbalanced.
BibTeX
@inproceedings{yu2022anisotropic,
  author = {Huangjie Yu and Anpei Chen and Xin Chen and Lan Xu and Ziyu Shao and Jingyi Yu},
  title = {{Anisotropic Fourier Features for Neural Image-Based Rendering and Relighting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20223},
}
Figure