Abstract
This paper presents a novel hybrid 3D representation that integrates textured meshes with 3D Gaussian Splatting (3DGS) to optimize indoor scene reconstruction. By assigning flat, texture-heavy regions to efficient meshes and complex geometries to Gaussian primitives, the proposed method achieves significant reductions in Gaussian counts and substantial improvements in rendering speed without compromising visual fidelity.
Introduction
3D Gaussian Splatting (3DGS) has emerged as a powerful tool for real-time 3D reconstruction. However, it suffers from a "brute force" inefficiency when representing flat surfaces with intricate textures, such as patterned rugs or wallpaper. In these cases, 3DGS generates an excessive number of tiny primitives to capture color variations, creating a rendering bottleneck. Conversely, while meshes are highly efficient for flat surfaces, they fail to capture complex, non-manifold geometries like foliage or thin wires. This work bridges the gap by creating a system where meshes and Gaussians play to their respective strengths.
Methodology
The proposed pipeline consists of three core stages: mesh extraction and pruning, joint rasterization, and transmittance-aware supervision.
- Mesh Extraction and Pruning: The system extracts an initial mesh and prunes it in geometrically complex areas using metrics like normal map variation (aided by StableNormal), face angles, and triangle size. This removes the mesh from difficult regions, leaving "holes" for Gaussians to fill.
- Joint Rasterization: A differentiable rendering pipeline integrates both representations. The textured mesh acts as an opaque background; during splatting, Gaussians are only rendered if they are positioned in front of the mesh.
- Transmittance-Aware Supervision: To prevent "ghosting"—where foreground colors bleed into background textures—the authors introduce a transmittance-aware texture loss. Using a sigmoid-based mask, the system determines if a pixel is occluded by Gaussians; if so, mesh texture optimization for that pixel is suppressed.
Key Results
Evaluation on datasets like Deep Blending, ScanNet++, and Replica demonstrates that the hybrid approach reduces Gaussian primitives by 18% to 35% in real-world scenes, and up to 50% when combined with other pruning methods.
Rendering performance on the Deep Blending dataset increased significantly from 346 FPS to 498 FPS. Despite the reduced primitive count, the model maintains high visual fidelity, often providing sharper details on flat surfaces where pure 3DGS can appear slightly blurry.
Conclusion and Future Work
The hybrid mesh-Gaussian representation offers a scalable solution for efficient indoor reconstruction by offloading high-frequency textures on flat surfaces to meshes. While currently limited by initial mesh quality and static scene assumptions, future work may involve optimizing vertex coordinates during training or extending the method to dynamic environments.
Summary updated