StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering

Lukas Radl, Michael Steiner, Mathias Parger, Alexander Weinrauch, Bernhard Kerbl, Markus Steinberger

TL;DR

This paper introduces "StopThePop," a novel hierarchical rendering approach for Gaussian Splatting that eliminates annoying "popping" artifacts during camera movement. By intelligently sorting and culling 3D Gaussians in a view-consistent manner, it achieves significantly improved visual quality without a major performance penalty, even outperforming the original method in speed and memory usage.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time 3D scene rendering by representing scenes as collections of 3D Gaussians. However, a key simplification in its rendering pipeline involves projecting these 3D Gaussians into 2D "splats" and using a simplified depth sorting mechanism. This approach, while fast, leads to "popping" artifacts – sudden, jarring changes in appearance – when the camera rotates. These artifacts break immersion, especially in interactive applications like virtual reality. The challenge is to achieve accurate per-pixel depth sorting for view consistency without incurring the prohibitive computational cost of a full, brute-force sort.

Methods

The core of StopThePop is a hierarchical rasterization approach that systematically refines the sorting of Gaussian splats. Instead of a single global sort, it employs a multi-stage process:

  • Tile-based Culling: Gaussians are first filtered to only consider those that could potentially contribute to specific image tiles, significantly reducing the number of splats to process.
  • Per-tile Depth Adjustment: A more accurate depth estimation (topt) is computed for each Gaussian within a tile, considering the point of maximum contribution along the view ray, rather than a simplified projection.
  • Hierarchical Sorting: The rendering pipeline is structured to mirror the GPU's memory and execution hierarchy. It uses multiple levels of sorted queues (4x4 tiles, 2x2 tiles, and per-pixel) to progressively refine the sort order. This allows for efficient sharing of sorting efforts between neighboring rays and early ray termination.
  • Load Balancing: A two-stage load balancing scheme ensures efficient distribution of work across GPU threads, particularly for large Gaussians that might otherwise bottleneck the pipeline.
  • Backward Pass: The method also implements a front-to-back blending order for the backward pass (gradient computation), which is more efficient and avoids the memory overhead of storing per-pixel sorted Gaussians.

Datasets

The paper evaluates its method on several standard datasets for 3D scene reconstruction and rendering:

  • Mip-NeRF 360: Both indoor and outdoor scenes.
  • Tanks & Temples: A benchmark for large-scale scene reconstruction.
  • Deep Blending: A dataset for free-viewpoint image-based rendering.

The authors also mention using COLMAP for sparse point cloud initialization, which is a standard practice in 3DGS.

Key Insights and Ideas

  • Popping is a consequence of approximate depth sorting: The paper clearly demonstrates that the simplified global depth sorting in 3DGS is the root cause of popping artifacts, especially during rotations.
  • Per-ray depth is crucial for consistency: Calculating the optimal depth (topt) for each Gaussian along a specific view ray, rather than using a global approximation, is key to eliminating popping.
  • Hierarchical processing mirrors GPU architecture: By structuring the sorting and culling process to align with the GPU's warp and thread hierarchy, significant performance gains can be achieved while maintaining accuracy.
  • "Cheating" with popping can improve image metrics: The paper suggests that 3DGS might implicitly "cheat" by allowing popping to occur, which can sometimes lead to better scores in standard image quality metrics, but at the cost of view consistency.
  • Reduced Gaussian count can exacerbate popping in naive methods: Enabling opacity decay to reduce the number of Gaussians leads to significantly more popping in 3DGS, highlighting the robustness of StopThePop.

Key Results

  • Elimination of Popping Artifacts: Qualitatively and quantitatively, StopThePop effectively removes popping artifacts, leading to much smoother and more immersive rendering, especially during camera rotations.
  • Comparable Image Quality: The method achieves image quality (PSNR, SSIM, LPIPS) comparable to or even slightly better than 3DGS, particularly in scenarios where 3DGS relies on popping.
  • Improved View Consistency: Metrics like FLIP show a significant improvement in view consistency, with the method outperforming 3DGS by a large margin, especially for longer temporal distances (FLIP7).
  • Performance Gains: Despite the added complexity, StopThePop is only about 4% slower on average than original 3DGS. Furthermore, by enforcing consistency, it allows for a reduction in the number of Gaussians by approximately half, leading to a 1.6x speedup and a 50% reduction in memory requirements compared to 3DGS in that configuration.
  • User Study Validation: A user study with 18 participants showed a clear preference for StopThePop, with participants rating it as more consistent in 54.3% of cases.

Limitations and Future Work

  • Resorting doesn't guarantee perfect order: While significantly improved, the hierarchical resorting process doesn't guarantee a perfectly sorted order in all edge cases, potentially leading to minor popping or flickering in very complex scenarios.
  • Ignoring Gaussian overlaps: The method still approximates 3D Gaussians and doesn't fully account for overlaps between them along the view ray.
  • Future Work: The authors suggest that a fully correct volume rendering of Gaussians, which explicitly handles overlaps, could further eliminate artifacts and lead to even better scene reconstructions. They also mention that their renderer and optimizations are publicly available.

Summary updated

Abstract

Gaussian Splatting has emerged as a prominent model for constructing 3D representations from images across diverse domains. However, the efficiency of the 3D Gaussian Splatting rendering pipeline relies on several simplifications. Notably, reducing Gaussian to 2D splats with a single viewspace depth introduces popping and blending artifacts during view rotation. Addressing this issue requires accurate per-pixel depth computation, yet a full per-pixel sort proves excessively costly compared to a global sort operation. In this paper, we present a novel hierarchical rasterization approach that systematically resorts and culls splats with minimal processing overhead. Our software rasterizer effectively eliminates popping artifacts and view inconsistencies, as demonstrated through both quantitative and qualitative measurements. Simultaneously, our method mitigates the potential for cheating view-dependent effects with popping, ensuring a more authentic representation. Despite the elimination of cheating, our approach achieves comparable quantitative results for test images, while increasing the consistency for novel view synthesis in motion. Due to its design, our hierarchical approach is only 4% slower on average than the original Gaussian Splatting. Notably, enforcing consistency enables a reduction in the number of Gaussians by approximately half with nearly identical quality and view-consistency. Consequently, rendering performance is nearly doubled, making our approach 1.6x faster than the original Gaussian Splatting, with a 50% reduction in memory requirements. Our renderer is publicly available at https://github.com/r4dl/StopThePop.
BibTeX
@article{radl2024stopthepop,
  author = {Lukas Radl and Michael Steiner and Mathias Parger and Alexander Weinrauch and Bernhard Kerbl and Markus Steinberger},
  title = {{StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering}},
  journal = {ACM Transactions on Graphics},
  year = {2024},
  doi = {10.1145/3658187},
}

Loading related content...

Figure