Abstract
We present PRTGaussian, a realtime relightable novel-view synthesis method made possible by combining 3D Gaussians and Precomputed Radiance Transfer (PRT). By fitting relightable Gaussians to multi-view OLAT data, our method enables real-time, free-viewpoint relighting. By estimating the radiance transfer based on high-order spherical harmonics, we achieve a balance between capturing detailed relighting effects and maintaining computational efficiency. We utilize a two-stage process: in the first stage, we reconstruct a coarse geometry of the object from multi-view images. In the second stage, we initialize 3D Gaussians with the obtained point cloud, then simultaneously refine the coarse geometry and learn the light transport for each Gaussian. Extensive experiments on synthetic datasets show that our approach can achieve fast and high-quality relighting for general objects.
Introduction
Creating digital objects that support both free-viewpoint navigation and dynamic relighting is a fundamental challenge in computer graphics. This task, often framed as inverse rendering, requires disentangling an object's geometry and material properties from its interaction with incident light.
Existing solutions often face a trade-off between speed and quality. Implicit representations like Neural Radiance Fields (NeRF) produce high-fidelity results but suffer from slow training and rendering due to dense sampling along rays. While some methods utilize One-Light-At-a-Time (OLAT) datasets to capture complex light transport effects like subsurface scattering and soft shadows, they often result in models too computationally expensive for real-time applications like VR or gaming.
Methodology
PRTGaussian addresses these limitations by combining the efficiency of 3D Gaussian Splatting (3DGS) with the mathematical framework of Precomputed Radiance Transfer (PRT).
3D Gaussian Splatting and PRT
Instead of a continuous volume, the scene is represented by millions of 3D ellipsoids (Gaussians). These are explicit primitives that can be projected (splatted) onto a 2D screen extremely quickly. To handle lighting, the method assigns a "transfer vector" to every Gaussian, describing how that specific point interacts with light.
High-Order Spherical Harmonics
Both incident light and learned transfer vectors are encoded using Spherical Harmonics (SH). While standard 3DGS uses low-order SH for color, PRTGaussian employs 9th-order SH. This higher order is critical for capturing high-frequency details such as shadows and inter-reflections.
Two-Stage Training Process
The training is split into two distinct phases to stabilize the learning of geometry and light transport:
- Geometry Reconstruction: A coarse geometry is first established using standard 3DGS on images captured under uniform lighting.
- Light Transport Learning: Gaussians are initialized from the coarse point cloud. A neural network (MLP) then regresses radiance transfer coefficients based on each Gaussian's position while the geometry is jointly refined.
Experiments and Results
The method was evaluated on synthetic multi-view OLAT datasets generated in Blender, featuring objects like a Chair, Microphone, and Hotdog.
Quantitative Performance
PRTGaussian demonstrates significant improvements over field-based methods like NRHints:
- Rendering Speed: Achieves ~333 FPS (0.0030s per frame), a ~5,900x speedup over NRHints (17.95s).
- Training Efficiency: Training completes in 15 minutes on a single RTX 4090, compared to 28 hours for NRHints.
- Visual Fidelity: Achieves a PSNR of 33.63, outperforming NRHints' 30.77.
Qualitative Comparison
As shown in Figure 2, PRTGaussian successfully recovers fine geometric details and complex self-shadowing that previous methods struggle with, such as the thin wires of the microphone.
Conclusion and Limitations
PRTGaussian enables real-time, high-quality relighting by leveraging explicit 3D Gaussian representations and precomputed radiance transfer. While highly efficient, the current model is limited to diffuse materials and struggles with perfectly sharp "hard" shadows due to the smooth nature of Spherical Harmonics. Future work includes extending the pipeline to handle specular highlights and incorporating visibility modeling to sharpen shadow edges.
Summary updated