Abstract
Precomputed Radiance Transfer (PRT) remains an attractive solution for real-time rendering of complex light transport effects such as glossy global illumination. After precomputation, we can relight the scene with new environment maps while changing viewpoint in real-time. However, practical PRT methods are usually limited to low-frequency spherical harmonic lighting. All-frequency techniques using wavelets are promising but have so far had little practical impact. The curse of dimensionality and much higher data requirements have typically limited them to relighting with fixed view or only direct lighting with triple product integrals. In this paper, we demonstrate a hybrid neural-wavelet PRT solution to high-frequency indirect illumination, including glossy reflection, for relighting with changing view. Specifically, we seek to represent the light transport function in the Haar wavelet basis. For global illumination, we learn the wavelet transport using a small multi-layer perceptron (MLP) applied to a feature field as a function of spatial location and wavelet index, with reflected direction and material parameters being other MLP inputs. We optimize/learn the feature field (compactly represented by a tensor decomposition) and MLP parameters from multiple images of the scene under different lighting and viewing conditions. We demonstrate real-time (512 x 512 at 24 FPS, 800 x 600 at 13 FPS) precomputed rendering of challenging scenes involving view-dependent reflections and even caustics.
Introduction
In computer graphics, achieving realistic global illumination in real-time remains a significant challenge. Precomputed Radiance Transfer (PRT) has long been a preferred solution, allowing for the "baking" of light transport within static scenes. This enables users to change lighting conditions and viewpoints instantly at runtime. However, traditional PRT is often hamstrung by the "curse of dimensionality." To accurately render glossy reflections that change with the viewer's perspective (view-dependence) under sharp, high-frequency lighting, one must account for a 6D light transport tensor—covering 3D spatial positions, 2D light directions, and 2D view directions.
Storing and processing this raw data is computationally prohibitive. Previous attempts to solve this have either relied on Spherical Harmonics, which result in blurry, low-frequency lighting, or restricted the user to a fixed viewpoint. This paper introduces a hybrid neural-wavelet framework designed to provide all-frequency, free-viewpoint indirect illumination without the massive storage overhead typically associated with high-dimensional transport tensors.
Methodology
The proposed framework bridges the gap between traditional signal processing and modern deep learning through a four-pronged approach:
1. Haar Wavelet Representation
Rather than treating light transport as a black-box neural problem, the method projects environment lighting into a Haar wavelet basis. By maintaining the most significant 64 coefficients, the system can utilize a mathematically grounded dot product for the lighting integral at runtime. This hybrid approach ensures that the physics of light transport is preserved while maintaining computational efficiency.
2. Neural Transport Predictor
A compact Multi-Layer Perceptron (MLP) is employed to predict the wavelet transport coefficients. The MLP processes several key inputs: a spatial feature vector, the reflected view direction ($\omega_r$), the surface normal, and specific material parameters (BRDF). Using the reflected direction rather than a raw view direction significantly eases the learning process for the network, resulting in much sharper specular highlights.
3. Compact Feature Fields
To represent the scene's spatial complexity, the authors utilize a CP tensor decomposition. The spatial component is stored within a multiresolution hash grid (inspired by Instant NGP). This structure is highly efficient, allowing the model to capture fine geometric details and complex light paths—such as those required for caustics—without the mesh-density dependencies of vertex-based storage.
4. Importance-Based Training
Capturing rare but visually striking effects like caustics requires a specialized training strategy. The authors implemented an importance-based sampling method that focuses on high-variance areas and specular regions during the training phase, rather than relying on uniform pixel sampling. This ensures the model learns to prioritize the most difficult-to-render aspects of the scene.
Evaluation and Results
The system was evaluated using several complex 3D scenes, including the KITCHEN (glossy reflections), FOUR ANIMALS (caustics), ARMADILLO (color bleeding), and PBRT BATHROOM (near-mirror reflections). Training data consisted of approximately 4,000 ground-truth images per scene, rendered via an OptiX path tracer across 3,000 lighting conditions and 2,000 camera trajectories.
Performance and Quality
- Real-Time Rendering: The system achieves 24 FPS at 512x512 resolution on an NVIDIA RTX 4090.
- Visual Fidelity: The method significantly outperforms existing Neural PRT (NPRT) baselines. In the KITCHEN scene, the model achieved a PSNR of ~37.3 dB, compared to 33.0 dB for NPRT. It successfully reconstructs view-dependent caustics and glossy inter-reflections that were previously considered too expensive for real-time applications.
- Efficiency: The entire representation for a complex scene is compressed into approximately 113 MB, orders of magnitude smaller than an explicit 6D transport tensor.
Discussion and Limitations
While the hybrid neural-wavelet approach represents a significant step forward, certain limitations remain. The method handles glossy surfaces exceptionally well but still struggles with perfect mirror reflections; the current limit of 64 wavelet coefficients is insufficient to prevent blurring in near-perfect reflectors. Additionally, the system is designed for static geometry; any movement of objects within the scene would require a full re-computation of the transport field. Finally, training time remains a factor, with a single scene requiring roughly 16 hours on an RTX 3090 Ti.
Conclusion
By combining the mathematical rigor of Haar wavelets with the compression capabilities of multiresolution hash grids and MLPs, this research provides a viable path for all-frequency, free-viewpoint relighting. The results demonstrate that preserving the linearity of light transport through a hybrid approach yields higher quality and more physically accurate results than purely neural alternatives, paving the way for more realistic real-time rendering in games and interactive simulations.
Summary updated