Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction

Binxiao Huang, Zhihao Li, Shiyong Liu, Xiao Tang, Jiajun Tang, Jiaqi Lin, Yuxin Cheng, Zhenyu Chen, Xiaofei Wu, Ngai Wong

Abstract

3D Gaussian splatting (3DGS) has demonstrated exceptional performance in image-based 3D reconstruction and real-time rendering. However, regions with complex textures require numerous Gaussians to capture significant color variations accurately, leading to inefficiencies in rendering speed. To address this challenge, we introduce a hybrid representation for indoor scenes that combines 3DGS with textured meshes. Our approach uses textured meshes to handle texture-rich flat areas, while retaining Gaussians to model intricate geometries. The proposed method begins by pruning and refining the extracted mesh to eliminate geometrically complex regions. We then employ a joint optimization for 3DGS and mesh, incorporating a warm-up strategy and transmittance-aware supervision to balance their contributions seamlessly. Extensive experiments demonstrate that the hybrid representation maintains comparable rendering quality and achieves superior frames per second (FPS) with fewer Gaussian primitives.

Introduction

In the realm of 3D reconstruction, 3D Gaussian Splatting (3DGS) has emerged as a leading technique for high-quality, real-time rendering. However, it faces a significant efficiency bottleneck: representing high-frequency textures on flat surfaces—such as detailed wallpaper or rugs—often requires a massive number of tiny Gaussian primitives. This redundancy is computationally expensive, as rendering millions of overlapping dots is more taxing than rendering a single textured triangle. Conversely, traditional meshes struggle with complex, non-manifold geometries like wires or plants. This work introduces a hybrid approach that leverages the efficiency of meshes for flat surfaces and the detail of Gaussians for intricate structures.

Methodology

The proposed pipeline integrates 3DGS and textured meshes through a multi-stage optimization process:

  1. Mesh Extraction and Pruning: A raw mesh is initially extracted from the scene. To ensure the mesh only represents reliable, flat regions, a pruning strategy is applied based on metrics such as normal variation (using a pre-trained StableNormal model), face angles, and triangle size. This removes the mesh from geometrically complex areas, leaving them to be handled by Gaussians.
  2. Joint Rasterization: A differentiable renderer was developed to handle both representations simultaneously. The mesh serves as an opaque background with fixed depth, while Gaussians are "splatted" in front. The final pixel color is a blend of Gaussian and mesh contributions, governed by the Gaussians' transmittance.
  3. Transmittance-Aware Supervision: To prevent "ghosting" artifacts—where foreground objects are incorrectly baked into background textures—the authors introduced a transmittance-aware mask ($M_T$). This mask ensures the mesh is only supervised by the ground truth image when the foreground Gaussians are sufficiently transparent, preventing texture bleeding.
Overview of the hybrid representation pipeline, showing the transition from prune metrics to textured mesh and final hybrid rendering.

Key Results and Discussion

The hybrid approach yields significant gains in efficiency without sacrificing visual fidelity. On the Deep Blending dataset, the method achieved 498 FPS, a substantial improvement over the 346 FPS of standard 3DGS. Furthermore, the number of Gaussian primitives was reduced by 18% to 35%, and up to 50% when combined with additional pruning techniques.

Qualitative comparison between vanilla 3D GS and the proposed hybrid representation, highlighting improved detail and reduced artifacts in the 'Ours' column.

Qualitatively, the mesh-based areas appear sharper and more stable than Gaussian-only representations, which can suffer from blurriness on flat surfaces. Quantitatively, the method maintains parity with state-of-the-art techniques in SSIM, PSNR, and LPIPS metrics across standard benchmarks like ScanNet++ and Replica.

Conclusion

By delegating flat, textured tasks to meshes and complex geometries to Gaussians, this hybrid representation optimizes the strengths of both methods. While currently most effective in indoor environments with prominent flat surfaces, future work will explore optimizing mesh vertex positions during joint training and integrating more complex material models to better capture view-dependent effects like specular highlights.

Summary updated

Abstract

3D Gaussian splatting (3DGS) has demonstrated exceptional performance in image-based 3D reconstruction and real-time rendering. However, regions with complex textures require numerous Gaussians to capture significant color variations accurately, leading to inefficiencies in rendering speed. To address this challenge, we introduce a hybrid representation for indoor scenes that combines 3DGS with textured meshes. Our approach uses textured meshes to handle texture-rich flat areas, while retaining Gaussians to model intricate geometries. The proposed method begins by pruning and refining the extracted mesh to eliminate geometrically complex regions. We then employ a joint optimization for 3DGS and mesh, incorporating a warm-up strategy and transmittance-aware supervision to balance their contributions seamlessly.Extensive experiments demonstrate that the hybrid representation maintains comparable rendering quality and achieves superior frames per second FPS with fewer Gaussian primitives.
BibTeX
@inproceedings{huang2025hybrid,
  author = {Binxiao Huang and Zhihao Li and Shiyong Liu and Xiao Tang and Jiajun Tang and Jiaqi Lin and Yuxin Cheng and Zhenyu Chen and Xiaofei Wu and Ngai Wong},
  title = {{Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/128},
}
Figure