Introduction
3D Gaussian Splatting (3DGS) has fundamentally changed the landscape of real-time 3D reconstruction and novel view synthesis. However, despite its rendering efficiency, the training process remains a significant temporal bottleneck. Standard 3DGS implementations rely on first-order Stochastic Gradient Descent (SGD), which typically requires tens of thousands of iterations and 10–30 minutes of compute time to achieve high-fidelity results.
3DGS$^2$ introduces a "stochastic local Newton" method that accelerates this training process by 5× to 10×. By exploiting the unique mathematical properties of Gaussian kernels—specifically the "weak coupling" of their attributes—the authors replace slow, linear-converging gradient descent with a near second-order convergent algorithm. This allows for high-quality reconstructions in seconds rather than minutes.
The Challenge: Beyond First-Order Optimization
In deep learning and 3D reconstruction, SGD is the industry standard due to its simplicity and scalability. However, its linear convergence rate is inherently limited. Second-order methods, such as Newton’s method, offer quadratic convergence, which could theoretically reach the same loss levels in a fraction of the steps.
In the context of 3DGS, a standard Newton method is traditionally impossible. Calculating a "global Hessian" (a matrix of second derivatives) for millions of Gaussians is computationally ruinous in terms of both memory and processing power. 3DGS$^2$ overcomes this by asking a pivotal question: Can we achieve the speed of Newton's method without the impossible overhead of a global Hessian?
Methodology: Stochastic Local Newton
The authors introduce three primary innovations to make second-order optimization practical for 3DGS:
1. Parallelized Local Newton Solve
The researchers observed that the influence of a Gaussian kernel is localized; a change in one kernel's attributes rarely impacts all others globally. This "weak coupling" allows the system to ignore the global Hessian. Instead, the algorithm analytically derives and solves small, independent Newton systems for each kernel's attribute groups (position, scale, rotation, opacity, and color) in parallel across GPU threads.
2. Secondary Targets for Stability
Second-order methods are notoriously prone to "overshooting" in stochastic settings where only one training image is viewed at a time. To stabilize the optimization, 3DGS$^2$ identifies Secondary Targets—the $K$ nearest neighboring camera views to the current training image. By incorporating a downsampled loss from these neighbors, the optimizer gains a "spatial look-ahead." This prevents aggressive updates from improving one view while inadvertently ruining the reconstruction from nearby angles.
3. Prioritized Optimization Order
The method enforces a specific sequence for attribute updates: Position $\rightarrow$ Shape (Rotation/Scale) $\rightarrow$ Appearance (Opacity/Color). This hierarchy reflects the physical reality of the scene: if a Gaussian is not correctly positioned in 3D space, adjusting its color or opacity is computationally futile.
Key Insights
- Hessian Sparsity: Visual analysis of the full Hessian in a 3DGS scene reveals that influence is highly localized. This "spiky" sparsity justifies the shift from global to local, per-kernel optimizations.
- Eliminating Learning Rates: Unlike SGD, which requires meticulous tuning of learning rates to prevent divergence, a valid quadratic approximation (Newton's method) naturally utilizes a step size of 1.0. This significantly simplifies the hyperparameter search for new datasets.
- Spatial Correlation as a Pre-conditioner: The use of neighboring views acts as a self-adjusting pre-conditioner, providing enough global context to keep the local Newton solver stable without the cost of processing the entire dataset at every iteration.
Experimental Results
The authors evaluated 3DGS$^2$ on standard benchmarks, including Mip-NeRF360, Tanks & Temples, and Deep Blender. The results demonstrate a paradigm shift in training efficiency:
- Speed: The method is consistently 5× to 10× faster than vanilla 3DGS. A scene that requires over 6 minutes to train with standard methods can be completed in just 46 seconds using 3DGS$^2$.
- Efficiency: The algorithm achieves superior or equal quality (measured by PSNR, SSIM, and LPIPS) using roughly one-tenth the iterations of gradient-based methods.
- Visual Fidelity: In the "Deep Blender" dataset, 3DGS$^2$ achieved a PSNR of 30.43, outperforming vanilla 3DGS (29.90) while maintaining a significantly faster training profile.
Limitations and Future Work
While 3DGS$^2$ represents a major leap forward, it is currently optimized for static scenes. In dynamic environments, the coupling between Gaussians changes over time, which may require further refinements to the local Newton approach. Additionally, while the method is fast, evaluating the Hessian for the position attribute remains the most computationally intensive part of the pipeline. Future work may involve integrating spatial data structures like AABBs or spatial hashing to further alleviate bottlenecks in kernel sorting.
Summary updated