Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction

Binxiao Huang, Zhihao Li, Shiyong Liu, Xiao Tang, Jiajun Tang, Jiaqi Lin, Yuxin Cheng, Zhenyu Chen, Xiaofei Wu, Ngai Wong

Abstract

This paper presents a novel hybrid 3D representation that integrates textured meshes with 3D Gaussian Splatting (3DGS) to optimize indoor scene reconstruction. By assigning flat, texture-heavy regions to efficient meshes and complex geometries to Gaussian primitives, the proposed method achieves significant reductions in Gaussian counts and substantial improvements in rendering speed without compromising visual fidelity.

Introduction

3D Gaussian Splatting (3DGS) has emerged as a powerful tool for real-time 3D reconstruction. However, it suffers from a "brute force" inefficiency when representing flat surfaces with intricate textures, such as patterned rugs or wallpaper. In these cases, 3DGS generates an excessive number of tiny primitives to capture color variations, creating a rendering bottleneck. Conversely, while meshes are highly efficient for flat surfaces, they fail to capture complex, non-manifold geometries like foliage or thin wires. This work bridges the gap by creating a system where meshes and Gaussians play to their respective strengths.

Methodology

The proposed pipeline consists of three core stages: mesh extraction and pruning, joint rasterization, and transmittance-aware supervision.

Overview of the hybrid pipeline, showing mesh pruning, joint rasterization, and the final hybrid rendering.
  1. Mesh Extraction and Pruning: The system extracts an initial mesh and prunes it in geometrically complex areas using metrics like normal map variation (aided by StableNormal), face angles, and triangle size. This removes the mesh from difficult regions, leaving "holes" for Gaussians to fill.
  2. Joint Rasterization: A differentiable rendering pipeline integrates both representations. The textured mesh acts as an opaque background; during splatting, Gaussians are only rendered if they are positioned in front of the mesh.
  3. Transmittance-Aware Supervision: To prevent "ghosting"—where foreground colors bleed into background textures—the authors introduce a transmittance-aware texture loss. Using a sigmoid-based mask, the system determines if a pixel is occluded by Gaussians; if so, mesh texture optimization for that pixel is suppressed.

Key Results

Evaluation on datasets like Deep Blending, ScanNet++, and Replica demonstrates that the hybrid approach reduces Gaussian primitives by 18% to 35% in real-world scenes, and up to 50% when combined with other pruning methods.

Qualitative comparison between vanilla 3DGS and the proposed hybrid method, highlighting the reduction in Gaussian primitives while maintaining detail.

Rendering performance on the Deep Blending dataset increased significantly from 346 FPS to 498 FPS. Despite the reduced primitive count, the model maintains high visual fidelity, often providing sharper details on flat surfaces where pure 3DGS can appear slightly blurry.

Conclusion and Future Work

The hybrid mesh-Gaussian representation offers a scalable solution for efficient indoor reconstruction by offloading high-frequency textures on flat surfaces to meshes. While currently limited by initial mesh quality and static scene assumptions, future work may involve optimizing vertex coordinates during training or extending the method to dynamic environments.

Summary updated

Abstract

3D Gaussian splatting (3DGS) has demonstrated exceptional performance in image-based 3D reconstruction and real-time rendering. However, regions with complex textures require numerous Gaussians to capture significant color variations accurately, leading to inefficiencies in rendering speed. To address this challenge, we introduce a hybrid representation for indoor scenes that combines 3DGS with textured meshes. Our approach uses textured meshes to handle texture-rich flat areas, while retaining Gaussians to model intricate geometries. The proposed method begins by pruning and refining the extracted mesh to eliminate geometrically complex regions. We then employ a joint optimization for 3DGS and mesh, incorporating a warm-up strategy and transmittance-aware supervision to balance their contributions seamlessly.Extensive experiments demonstrate that the hybrid representation maintains comparable rendering quality and achieves superior frames per second FPS with fewer Gaussian primitives.
BibTeX
@inproceedings{huang2025hybrid,
  author = {Binxiao Huang and Zhihao Li and Shiyong Liu and Xiao Tang and Jiajun Tang and Jiaqi Lin and Yuxin Cheng and Zhenyu Chen and Xiaofei Wu and Ngai Wong},
  title = {{Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/128},
}

Loading related content...

Figure