Multi-View Relighting with Mergeable Radiance Fields

Yinan Wang, Rohan Kadkol, Hyunjin Yoo, Jean-François Lalonde, James J. ClarkarXiv 2026 · arXiv

TL;DR

MergeableRF is a novel framework that enables high-quality relighting of NeRFs and 3D Gaussian Splatting (3DGS) models by directly merging their neural network parameters. By combining shared initialization with "merge-aware" training, the authors show that you can linearly combine appearance networks to render a scene under any target illumination in a single rendering pass.

What Problem Does This Paper Address?

Radiance field methods like NeRF and 3DGS have revolutionized 3D reconstruction, but they typically "bake" the lighting into the scene's appearance. If you want to change the lighting (relighting), you usually have to perform complex inverse rendering to estimate materials and geometry or use heavy generative models.

A fundamental property of light is linearity: an image under a mixture of lights is simply the weighted sum of images under each individual light. While we can easily blend images, we cannot easily blend neural networks. If you train two separate NeRFs on two different lights and average their weights, the resulting model will likely produce garbage. This is because neural networks are highly non-linear in their parameter space. MergeableRF bridges this gap, allowing us to leverage the linearity of light transport directly within the model's parameters.

Methods

The core idea of MergeableRF is to make the appearance networks of a radiance field "mergeable" by construction.

  1. Shared Geometry, Separate Appearance: The model uses a single, shared representation for geometry (e.g., a hash grid for NeRF or a set of Gaussians for 3DGS). However, it uses a separate, small Multi-Layer Perceptron (MLP) to represent the appearance under each basis light (a set of known, fixed lighting directions).
  2. Shared Initialization: To ensure the different appearance MLPs stay in a compatible region of the loss landscape, they are all initialized with the exact same random weights.
  3. Merge-Aware Training: This is the "secret sauce." During training, the system doesn't just learn to reconstruct individual basis lights. It also samples random combinations of lights, merges the corresponding MLP parameters linearly, and supervises the output against the ground-truth image blend. Because light transport is linear, this "blended" ground truth is available for free.
  4. Relighting via Parameter Merging: To relight a scene under a new HDR environment map, the map is first decomposed into coefficients for the basis lights. These coefficients are then used to create a single "merged" appearance MLP. Rendering the relit scene then requires only one rendering pass, regardless of how complex the target lighting is.
Figure 1. Mergeable radiance fields. (a) One radiance field, a NeRF or a set of 3D Gaussians, is shared by K basis illuminations Li. For every element j that contributes to a pixel (a sample along its ray for NeRF, a Gaussian that overlaps it for 3DGS), the field provides a feature zj and the viewing direction dj. Each basis light has its own small appearance MLP with parameters ϕi, trained on the images Ii of the scene under Li. All MLPs start from the same initialization. To make them mergeable, some of the training samples are rendered with a random convex combination ϕmix of the MLPs and supervised by the same combination of the basis images, which is the true image under the mixed light (bottom row). (b) To relight, we approximate a new environment map LA by a non-negative combination of the basis lights with weights βi, and take the same weighted sum of the MLP parameters, ϕM = P

Datasets

The authors evaluated their method on two primary datasets:

  • Neural Gaffer: A synthetic benchmark featuring various objects rendered under diverse HDR environment maps.
  • OLATverse: A real-world dataset consisting of multi-view captures of objects under "One-Light-At-a-Time" (OLAT) conditions in a light stage.

Key Insights and Ideas

  • Parameter-Space Linearity is Attainable: The paper demonstrates that while NNs are generally non-linear, they can be coaxed into supporting linear arithmetic if they share an initialization and are explicitly trained for it.
  • Efficiency through Merging: Traditional image-space blending requires rendering the scene $K$ times (once for each basis light) and then summing the images. MergeableRF reduces this to a single render, making it significantly more efficient for real-time applications.
  • Representation Agnostic: The method works equally well for both volumetric (NeRF) and point-based (3DGS) radiance fields, showing that the bottleneck for relighting is often the appearance model, not the underlying geometry representation.
  • The "Merge Gap": The authors identify a "merge gap"—the difference in quality between blending images vs. merging parameters. They show that their merge-aware training reduces this gap from a massive ~14.6 dB down to a negligible ~1.4 dB.

Key Results

MergeableRF achieves relighting quality that is competitive with state-of-the-art inverse rendering methods, but with much simpler machinery.

  • Quantitative: On the synthetic Neural Gaffer dataset, the NeRF-based MergeableRF achieved the second-highest PSNR (23.38 dB) among compared methods. On the real-world OLATverse data, the 3DGS-based version was the most accurate, matching or beating specialized point-light relighting methods.
  • Qualitative: The method successfully captures complex shading and global illumination effects. While it struggles with extremely sharp specular highlights (due to the smoothness of the basis lights), the overall color and shading are highly accurate.
Figure 2. Qualitative comparison on the Neural Gaffer objects. Each row shows an object relit under a new environment map (left) and seen from a held-out view. Every method uses the setting reported in Tab. 1. Our results come from a single render of the merged network, with no optimization for the new lighting. All methods handle the backpack well. The shiny helmet and iron are harder. Our models get the overall color and shading right but miss the sharp reflections, because our basis lights are smooth.

Limitations and Future Work

  • Basis Light Resolution: The current method uses relatively smooth basis lights (von Mises-Fisher lobes). This limits the model's ability to reproduce pin-sharp reflections from small, bright light sources.
  • Scaling the Basis: While increasing the number of basis lights improves the "upper bound" of lighting accuracy, it also makes the merging task harder for the networks. Finding the optimal balance or using hierarchical bases is a potential future direction.
  • Known Lighting: The method requires multi-view images under known basis lights for training, which typically necessitates a controlled environment like a light stage. Extending this to "in-the-wild" captures remains a challenge.

Summary updated

Abstract

We propose MergeableRF, a method for radiance field relighting through model parameter merging. Given multi-view images of a scene under known basis lights, our goal is to relight the scene by directly combining the parameters of radiance fields trained under these lights. This is motivated by the linearity of light transport: the image under a mixture of lights is the same weighted combination of the images under each light. However, combining the parameters of independently trained radiance fields does not produce the corresponding combination of images. To address this, we represent a scene with shared geometry and a separate appearance network for each basis light. We start these networks from the same initialization and introduce merge-aware training, which teaches their parameter combinations to reproduce the corresponding combinations of images. Given a target HDR environment map, we compute its basis-light coefficients and use them to merge the appearance networks into a single relit model. Our method supports both Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), without material estimation, additional relighting networks, or target-specific optimization. Unlike image-space blending, it requires only one rendering pass per viewpoint. Experiments on synthetic Neural Gaffer objects and real OLATverse captures demonstrate relighting quality competitive with methods trained on the same multi-light observations.
BibTeX
@misc{wang2026multi,
  author = {Yinan Wang and Rohan Kadkol and Hyunjin Yoo and Jean-François Lalonde and James J. Clark},
  title = {{Multi-View Relighting with Mergeable Radiance Fields}},
  year = {2026},
  eprint = {2611.25523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25523},
}
Figure