TL;DR
Gaussolve introduces a "subtract-and-refill" approach to hybrid scene rendering, combining the efficiency of textured meshes with the flexibility of 3D Gaussian Splatting (3DGS). By using a Neural Dissolve Field, the system selectively erases inconsistent mesh content and refills those gaps with residual Gaussians, achieving higher visual fidelity with significantly fewer primitives and faster rendering speeds.
What Problem Does This Paper Address?
While 3D Gaussian Splatting (3DGS) has revolutionized real-time novel-view synthesis, it is computationally expensive to represent large, flat surfaces (like walls or roads) using thousands of tiny Gaussian primitives. Hybrid methods attempt to solve this by using a textured mesh for the "heavy lifting" of regular geometry and Gaussians for the fine details.
However, existing hybrids are primarily additive: they treat the mesh as a fixed, opaque base and simply layer Gaussians on top. This creates a fundamental conflict. Meshes often contain errors—geometry misalignments, texture-projection artifacts, or incorrect occlusions—that vary depending on your viewing angle. If the mesh is wrong, simply adding Gaussians doesn't fix the underlying "bad" pixels. Furthermore, treating the mesh as a hard, opaque occluder blocks the "learning signal" (gradients) from reaching Gaussians behind the surface, making it nearly impossible for the model to correct errors.
Methods
The authors propose Gaussolve, which reinterprets the mesh-Gaussian conflict as a multi-view inconsistency problem. The framework consists of three main components:
- Textured Mesh Branch: Provides a compact structural prior for broad, regular regions.
- Neural Dissolve Field (NDF): This is the "subtraction" mechanism. It is parameterized by scalar spherical-harmonic (SH) Gaussians. Unlike standard Gaussians that emit color, these "dissolve Gaussians" predict a view-dependent value that determines how much of the mesh should be "erased" or attenuated from a specific direction.
- Residual RGB Gaussians: These "refill" the regions revealed by the NDF, handling fine details, thin structures, and complex view-dependent effects.
To make this work, the authors replace traditional "hard-Z" visibility (where a mesh is either 100% opaque or 0% there) with a Continuous Transmittance Fusion rule. This allows the mesh to become partially transparent, ensuring that reconstruction gradients can flow to the Gaussians behind it during training, enabling them to "learn" how to fill the gaps.
Datasets
The authors evaluated Gaussolve on several standard benchmarks for scene reconstruction:
- ML-Hypersim-AI: A dataset of highly structured indoor scenes.
- Deep Blending: A collection of semi-structured indoor and outdoor scenes.
- Mip-NeRF 360: Used for "stress tests" on unstructured outdoor scenes (like trees and bushes) to find the limits of the mesh-based approach.
Key Insights and Ideas
- Subtraction is as vital as addition: In hybrid rendering, the mesh isn't just a foundation; it's often a source of error. The ability to "dissolve" the mesh is what allows the Gaussians to actually do their job of correcting the scene.
- Errors are view-dependent: A mesh might look perfect from one angle but show a stretched texture or a "ghosting" artifact from another. By using scalar SH Gaussians for the NDF, Gaussolve can selectively hide the mesh only when viewed from problematic angles.
- Soft visibility solves "gradient starvation": Hard visibility boundaries are the enemy of differentiable rendering. By using a soft transmittance model, the system avoids the "Hard-Z" brittleness that prevents Gaussians from crossing the mesh boundary during optimization.
Key Results
Gaussolve demonstrates a superior quality-to-compactness trade-off compared to both pure 3DGS and previous hybrid methods:
- Visual Quality: Achieves higher PSNR and lower LPIPS (perceptual error) than baselines like MeshGS and 3D-GES. On the Deep Blending dataset, it improved average PSNR by ~0.7 dB over existing hybrids.
- Efficiency: Reduces the Gaussian primitive count by approximately 60% compared to standard 3DGS while maintaining or improving quality.
- Rendering Speed: Thanks to the reduced Gaussian count, it achieves significant speedups, reaching 174 FPS on an NVIDIA RTX 5000, compared to 99 FPS for standard 3DGS.
- Versatility: The framework is "backbone-agnostic," meaning it can be plugged into different Gaussian variants like 3DGS-MCMC or Reduced 3DGS to further enhance performance.
Limitations and Future Work
The primary limitation of Gaussolve is its reliance on a decent mesh prior. In highly unstructured scenes (like dense foliage or tangled branches), the initial mesh extraction often fails or produces a "soup" of triangles. In these cases, the mesh and Gaussian branches may compete to explain the same image data, leading to fewer efficiency gains.
The authors suggest that future work could involve geometry-aware routing, which would automatically decide which parts of a scene are best represented by a mesh versus Gaussians, even in highly irregular environments.
Summary updated