Abstract
We consider the problem of physically-based inverse rendering using 3D Gaussian Splatting (3DGS) representations. While recent 3DGS methods have achieved remarkable results in novel view synthesis (NVS), accurately capturing high-fidelity geometry, physically interpretable materials and lighting remains challenging, as it requires precise geometry modeling to provide accurate surface normals, along with physically-based rendering (PBR) techniques to ensure correct material and lighting disentanglement. Previous 3DGS methods resort to approximating surface normals, but often struggle with noisy local geometry, leading to inaccurate normal estimation and suboptimal material-lighting decomposition. In this paper, we introduce GeoSplatting, a novel hybrid representation that augments 3DGS with explicit geometric guidance and differentiable PBR equations. Specifically, we bridge isosurface and 3DGS together, where we first extract isosurface mesh from a scalar field, then convert it into 3DGS points and formulate PBR equations for them in a fully differentiable manner. In GeoSplatting, 3DGS is grounded on the mesh geometry, enabling precise surface normal modeling, which facilitates the use of PBR frameworks for material decomposition. This approach further maintains the efficiency and quality of NVS from 3DGS while ensuring accurate geometry from the isosurface. Comprehensive evaluations across diverse datasets demonstrate the superiority of GeoSplatting, consistently outperforming existing methods both quantitatively and qualitatively.
Introduction
Physically-based inverse rendering is a fundamental challenge in 3D vision, involving the decomposition of images into shape, material properties (e.g., roughness, albedo), and environmental lighting. While 3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis with its speed and quality, it inherently lacks a defined surface. This makes calculating accurate surface normals or modeling light transport (like shadows and inter-reflections) difficult. Existing 3DGS methods often produce noisy materials due to flawed geometric estimations, leading to suboptimal relighting results.
GeoSplatting addresses these issues by introducing explicit geometry guidance. By grounding Gaussian primitives onto an optimizable mesh surface, the method achieves state-of-the-art material decomposition and relighting quality with high efficiency.
Methodology
The GeoSplatting pipeline: from FlexiCubes isosurface extraction to mesh-grounded Gaussian points and PBR rendering. Source, p. 3
The GeoSplatting framework integrates explicit geometry into the 3DGS pipeline through several core components:
- MGadapter (Mesh-to-Gaussian Adaptor): Instead of free-floating Gaussians, GeoSplatting extracts an intermediate triangular mesh using FlexiCubes. The MGadapter binds Gaussian points directly to the mesh faces. As the mesh deforms during optimization, the Gaussians move accordingly, ensuring a mathematically well-defined surface and normal.
- Occlusion-Aware Light Transport: The explicit mesh allows for high-speed BVH-accelerated ray tracing to calculate self-occlusions (shadows), providing more accurate results than volumetric approximations.
- Physically-Based Rendering (PBR) Pipeline: The system utilizes the GGX microfacet model. Training follows a two-stage strategy: a fast "split-sum" approximation for initialization, followed by rigorous Monte Carlo sampling to capture complex light interactions.
- Deferred Shading Refinement: To capture high-frequency details like sharp reflections, a deferred shading pass performs PBR calculations at the pixel level.
Results and Evaluation
Qualitative comparison of GeoSplatting against baseline methods (GS-IR, R3DG) and ground truth for material decomposition and relighting. Source, p. 1
GeoSplatting was evaluated across diverse benchmarks, including Synthetic4Relight, TensoIR Synthetic, Shiny Blender, and real-world datasets like Stanford-ORB.
- Efficiency: GeoSplatting converges in approximately 14 minutes on a single NVIDIA RTX 4090, orders of magnitude faster than traditional implicit-field approaches like NeRO, which can take over 13 hours.
- Relighting Quality: On the Synthetic4Relight dataset, GeoSplatting achieved a 34.10 PSNR, significantly outperforming previous 3DGS-based methods such as GS-IR (23.81) and R3DG (31.00).
- Geometric Accuracy: The Mean Angular Error (MAE) for surface normals was reduced to 2.15 on the Shiny Blender dataset, compared to 4.4–7.0 for prior 3DGS methods, indicating superior shape recovery.
Conclusion
GeoSplatting successfully bridges the gap between the rendering efficiency of 3DGS and the geometric precision of triangular meshes. By anchoring Gaussians to a mesh, it enables accurate normal estimation and efficient light transport modeling. Future work will focus on overcoming resolution constraints of the isosurface grid and reducing the dependency on object masks for foreground separation.
Summary updated