Abstract
Recent reconstruction methods based on radiance fields, such as NeRF and 3DGS, reproduce indoor scenes with high visual fidelity but fail during scene editing due to baked illumination and a lack of explicit light transport. Physically based inverse rendering typically relies on mesh representations and path tracing, which enforce correct light transport but are limited by the geometric fidelity of reconstructed meshes. We propose Emission-Aware Gaussians and Path Tracing (EAG-PT), a framework for physically based light transport using a unified 2D Gaussian representation. By explicitly separating emissive and non-emissive components and decoupling reconstruction from multi-bounce path tracing, EAG-PT produces physically consistent renders after editing while preserving fine geometric details.
Introduction: The Challenge of Baked Illumination
Modern 3D reconstruction techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) excel at novel view synthesis but struggle with scene editing. Because these methods "bake" the lighting directly into the scene's appearance, modifying the environment—such as changing a light's color or moving an object—results in unrealistic visuals that lack proper shadows and interreflections.
Traditional solutions involve inverse rendering to extract a 3D mesh for use in path tracing. However, extracting high-quality meshes from cluttered real-world scenes is a significant bottleneck. Errors in the mesh geometry, such as jagged edges on thin structures, lead to severe artifacts in light simulation. EAG-PT addresses this by asking: Can we perform physically accurate path tracing directly on a Gaussian representation, bypassing the need for meshes entirely?
Methodology: Mesh-Free Path Tracing with 2D Gaussians
EAG-PT introduces a pipeline that bypasses the mesh bottleneck by performing path tracing directly on 2D Gaussians. Unlike their 3D counterparts, 2D Gaussians act as flat surface elements with well-defined normals, making them ideal proxies for ray-surface intersections.
The framework operates in three distinct phases:
- Radiant Scene Reconstruction: The scene is reconstructed using 2D Gaussians. Crucially, emissive light sources are explicitly masked and separated from non-emissive surfaces (e.g., walls and furniture).
- Material Recovery: To determine the true diffuse albedo of surfaces, the system uses a single-bounce differentiable renderer. It leverages the initial radiant field as a "radiance cache" to approximate incoming light efficiently during training, avoiding the cost of multi-bounce simulations.
- Path Tracing and Light Baking: After a user edits the scene, EAG-PT performs multi-bounce path tracing on the 2D Gaussians to simulate global illumination. To enable real-time interaction, the path-traced results are "baked" back into the Gaussians' radiance parameters, dropping render times from minutes to milliseconds.
Results and Discussion
EAG-PT significantly outperforms standard radiant scene reconstructions in editing tasks. While basic 3DGS-based editing fails to account for light transport, EAG-PT captures complex physical effects like soft shadows and color bleeding (interreflection).
In comparisons with state-of-the-art mesh-based methods like FIPT, EAG-PT demonstrates superior preservation of fine geometric details. Thin structures like lamp arms and chair legs, which often become distorted in mesh reconstructions, remain sharp and visually accurate. Furthermore, the light baking strategy reduces rendering times from minutes per frame to just 0.015 seconds, enabling real-time playback at over 60 FPS with virtually no loss in visual quality.
Conclusion
EAG-PT provides a robust framework for physically accurate indoor scene editing without the need for error-prone mesh reconstruction. By combining the flexibility of 2D Gaussians with the physical rigor of path tracing, it opens new possibilities for interior design, XR content creation, and embodied AI. Future work will focus on extending the model to handle complex non-diffuse materials and automating the emission masking process.
Summary updated