Abstract
Decoupling the illumination in 3D scenes is crucial for novel view synthesis and relighting. In this paper, we propose a novel method for representing a scene illuminated by a point light using a set of relightable 3D Gaussian points. Inspired by the Blinn-Phong model, our approach decomposes the scene into ambient, diffuse, and specular components, enabling the synthesis of realistic lighting effects. To facilitate the decomposition of geometric information independent of lighting conditions, we introduce a novel bilevel optimization-based meta-learning framework. The fundamental idea is to view the rendering tasks under various lighting positions as a multi-task learning problem, which our meta-learning approach effectively addresses by generalizing the learned Gaussian geometries not only across different viewpoints but also across diverse light positions. Experimental results demonstrate the effectiveness of our approach in terms of training efficiency and rendering quality compared to existing methods for free-viewpoint relighting.
Introduction
Relighting is a fundamental challenge in 3D reconstruction, aiming to render objects under arbitrary lighting conditions from a limited set of observations. While recent advances like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have achieved impressive results in novel view synthesis, they often struggle with "Out-of-Distribution" (OOD) relighting. In these scenarios, the lighting at test time differs significantly from the training data, leading to "baked-in" shadows and highlights that fail to move realistically with the light source. This is particularly evident in "One Light At a Time" (OLAT) setups, where models often overfit to specific training views rather than learning the underlying physical properties of the scene.
Methodology
The proposed framework, MetaGS, addresses these challenges through two key innovations: a meta-learning optimization strategy and a physically-based reflection model.
- Meta-Learning Framework: MetaGS employs a bilevel optimization approach, treating different lighting conditions as distinct tasks. By utilizing an inner loop for adaptation and an outer loop for generalization, the model is forced to learn a consistent geometry that remains valid across diverse light positions. This prevents the optimization from creating "floaters" or artifacts to explain away complex lighting effects.
- Differentiable Blinn-Phong Model: Each Gaussian point is equipped with a learnable Blinn-Phong reflection model. This explicitly decomposes the appearance into ambient, diffuse, and specular components. By constraining the appearance to follow physical laws, the model effectively separates geometry (normals) from material properties (reflectance).
- Explicit Shadow Mapping: To handle occlusions, the system incorporates a Bounding Volume Hierarchy (BVH)-based ray tracing method. This allows for the rendering of physically plausible shadows by explicitly calculating the visibility between Gaussian centers and the light source.
Experimental Results
MetaGS was evaluated on both synthetic (e.g., Layered Woven Ball) and real-world (NRHints dataset) scenes. In synthetic OOD tasks, MetaGS achieved a significant performance boost, reaching a PSNR of ~27.42 compared to ~19.14 for a non-meta-learned baseline. This represents a massive leap in image reconstruction quality for unseen lighting conditions.
Qualitatively, the method produces sharp, dynamic specular highlights and accurate shadows that correctly respond to changing light positions. The explicit decoupling of geometry and appearance allows the model to generalize to complex environment maps by summing contributions from multiple virtual point lights. Despite the complex optimization, the model remains efficient, training in roughly one hour on a single NVIDIA RTX 3090 GPU.
Conclusion
MetaGS demonstrates that integrating physical priors with meta-learning optimization significantly enhances the robustness of 3D Gaussian Splatting for relighting tasks. By forcing the model to follow the rules of physics, the system moves beyond simple pattern matching to achieve true geometric decoupling. While currently limited to direct illumination and basic material models, it provides a powerful foundation for future work in global illumination and complex material synthesis.
Summary updated