Abstract
Recent advances in 3D Gaussian Splatting (3DGS) have enabled high-quality real-time novel view synthesis, yet the entanglement of geometry and appearance often necessitates an excessive number of primitives to capture fine details. This overparameterization leads to geometric redundancy and visual artifacts, such as elongated "needle-like" shapes. We introduce Neural Shell Texture Splatting (NeST-Splatting), which decouples these components by representing appearance through a global, continuous neural texture field. By utilizing 2D surfels as spatial samplers for a multi-resolution hash encoding, our method achieves state-of-the-art visual fidelity using up to 3x fewer primitives. Experimental results across standard benchmarks demonstrate that NeST-Splatting significantly reduces visual artifacts and provides cleaner geometric reconstructions compared to existing splatting techniques.
Introduction
While 3D Gaussian Splatting (3DGS) has revolutionized real-time Novel View Synthesis (NVS), it suffers from a fundamental "entanglement" issue. In standard 3DGS, each Gaussian primitive stores its own color, typically via Spherical Harmonics. To represent surfaces with complex, high-frequency textures, the system must undergo aggressive "densification," creating a massive number of tiny primitives. This leads to significant redundancy—overparameterizing geometry to capture underparameterized texture—and creates visual artifacts. Specifically, Gaussians often shrink into tiny, elongated "needle" shapes to fit sharp texture edges, making it nearly impossible to extract clean 3D meshes.
Methodology
NeST-Splatting addresses these issues by separating the "where" (geometry) from the "what" (appearance). The method replaces 3D volumes with 2D oriented planar Gaussian disks, or surfels, which act as spatial samplers rather than independent color carriers.
As shown in the pipeline above, for every pixel, the system calculates the intersection between the camera ray and the 2D surfels. The 3D coordinates of these intersections are used to query a global Multi-resolution Hash Encoding (based on Instant-NGP), which serves as the "Neural Shell Texture." Instead of blending colors directly, the system alpha-blends the retrieved feature vectors into a screen-space feature map. A lightweight Multi-Layer Perceptron (MLP) then decodes this map, considering the view direction, to produce the final RGB image. This deferred rendering approach allows nearby primitives to share features from the same global hash grid, drastically increasing memory efficiency.
Key Insights and Ideas
- Disentanglement for Efficiency: By decoupling texture from primitives, the number of Gaussians required is no longer dictated by texture complexity. A simple geometric model can be covered in a highly complex "neural skin."
- Geometric Integrity: Because splats no longer need to deform to match texture boundaries, they maintain more balanced shapes, eliminating "needle" artifacts and facilitating easier mesh baking.
- Feature Level Annealing: To ensure stable optimization, the authors employ a strategy that gradually introduces higher-resolution texture details, allowing geometry and texture to optimize harmoniously.
Key Results
NeST-Splatting consistently outperforms baselines in perceptual quality, particularly in texture-rich regions. On the NeRFSynthetic dataset, it achieves higher visual quality using only 73k primitives compared to the 288k required by standard 3DGS.
As illustrated in the comparison above, NeST-Splatting maintains remarkable detail even when the primitive count is reduced to as low as 5,000 points. Quantitatively, the method excels in the LPIPS metric, which measures human-perceived image quality. On the DTU dataset, it achieved a Chamfer Distance of 0.83mm, comparable to state-of-the-art 2DGS but with 60% fewer points, proving that the reduction in primitives does not come at the cost of geometric accuracy.
Limitations and Future Work
The primary trade-off for this increased efficiency and quality is rendering speed. Querying the hash grid and running the MLP decoder for every frame reduces the frame rate to approximately 71 FPS on an RTX 4090, compared to the 224 FPS achievable by vanilla 2DGS. Training times are also longer, taking roughly 25 minutes. Future research will focus on closing this speed gap through hardware-accelerated neural decoding and more optimized kernel implementations.
Conclusion
NeST-Splatting represents a significant step forward in making Gaussian Splatting more efficient and geometrically sound. By treating Gaussians as samplers for a global neural texture field, the method achieves superior visual results with a fraction of the primitives, paving the way for more compact and accurate 3D digital twins.
Summary updated