TL;DR
MergeableRF is a novel framework that enables high-quality relighting of NeRFs and 3D Gaussian Splatting (3DGS) models by directly merging their neural network parameters. By combining shared initialization with "merge-aware" training, the authors show that you can linearly combine appearance networks to render a scene under any target illumination in a single rendering pass.
What Problem Does This Paper Address?
Radiance field methods like NeRF and 3DGS have revolutionized 3D reconstruction, but they typically "bake" the lighting into the scene's appearance. If you want to change the lighting (relighting), you usually have to perform complex inverse rendering to estimate materials and geometry or use heavy generative models.
A fundamental property of light is linearity: an image under a mixture of lights is simply the weighted sum of images under each individual light. While we can easily blend images, we cannot easily blend neural networks. If you train two separate NeRFs on two different lights and average their weights, the resulting model will likely produce garbage. This is because neural networks are highly non-linear in their parameter space. MergeableRF bridges this gap, allowing us to leverage the linearity of light transport directly within the model's parameters.
Methods
The core idea of MergeableRF is to make the appearance networks of a radiance field "mergeable" by construction.
- Shared Geometry, Separate Appearance: The model uses a single, shared representation for geometry (e.g., a hash grid for NeRF or a set of Gaussians for 3DGS). However, it uses a separate, small Multi-Layer Perceptron (MLP) to represent the appearance under each basis light (a set of known, fixed lighting directions).
- Shared Initialization: To ensure the different appearance MLPs stay in a compatible region of the loss landscape, they are all initialized with the exact same random weights.
- Merge-Aware Training: This is the "secret sauce." During training, the system doesn't just learn to reconstruct individual basis lights. It also samples random combinations of lights, merges the corresponding MLP parameters linearly, and supervises the output against the ground-truth image blend. Because light transport is linear, this "blended" ground truth is available for free.
- Relighting via Parameter Merging: To relight a scene under a new HDR environment map, the map is first decomposed into coefficients for the basis lights. These coefficients are then used to create a single "merged" appearance MLP. Rendering the relit scene then requires only one rendering pass, regardless of how complex the target lighting is.
Datasets
The authors evaluated their method on two primary datasets:
- Neural Gaffer: A synthetic benchmark featuring various objects rendered under diverse HDR environment maps.
- OLATverse: A real-world dataset consisting of multi-view captures of objects under "One-Light-At-a-Time" (OLAT) conditions in a light stage.
Key Insights and Ideas
- Parameter-Space Linearity is Attainable: The paper demonstrates that while NNs are generally non-linear, they can be coaxed into supporting linear arithmetic if they share an initialization and are explicitly trained for it.
- Efficiency through Merging: Traditional image-space blending requires rendering the scene $K$ times (once for each basis light) and then summing the images. MergeableRF reduces this to a single render, making it significantly more efficient for real-time applications.
- Representation Agnostic: The method works equally well for both volumetric (NeRF) and point-based (3DGS) radiance fields, showing that the bottleneck for relighting is often the appearance model, not the underlying geometry representation.
- The "Merge Gap": The authors identify a "merge gap"—the difference in quality between blending images vs. merging parameters. They show that their merge-aware training reduces this gap from a massive ~14.6 dB down to a negligible ~1.4 dB.
Key Results
MergeableRF achieves relighting quality that is competitive with state-of-the-art inverse rendering methods, but with much simpler machinery.
- Quantitative: On the synthetic Neural Gaffer dataset, the NeRF-based MergeableRF achieved the second-highest PSNR (23.38 dB) among compared methods. On the real-world OLATverse data, the 3DGS-based version was the most accurate, matching or beating specialized point-light relighting methods.
- Qualitative: The method successfully captures complex shading and global illumination effects. While it struggles with extremely sharp specular highlights (due to the smoothness of the basis lights), the overall color and shading are highly accurate.
Limitations and Future Work
- Basis Light Resolution: The current method uses relatively smooth basis lights (von Mises-Fisher lobes). This limits the model's ability to reproduce pin-sharp reflections from small, bright light sources.
- Scaling the Basis: While increasing the number of basis lights improves the "upper bound" of lighting accuracy, it also makes the merging task harder for the networks. Finding the optimal balance or using hierarchical bases is a potential future direction.
- Known Lighting: The method requires multi-view images under known basis lights for training, which typically necessitates a controlled environment like a light stage. Extending this to "in-the-wild" captures remains a challenge.
Summary updated