3DGS2: Near Second-order Converging 3D Gaussian Splatting

Lei Lan, Tianjia Shao, Zixuan Lu, Yu Zhang, Chenfanfu Jiang, Yin Yang

Introduction

3D Gaussian Splatting (3DGS) has fundamentally changed the landscape of real-time 3D reconstruction and novel view synthesis. However, despite its rendering efficiency, the training process remains a significant temporal bottleneck. Standard 3DGS implementations rely on first-order Stochastic Gradient Descent (SGD), which typically requires tens of thousands of iterations and 10–30 minutes of compute time to achieve high-fidelity results.

3DGS$^2$ introduces a "stochastic local Newton" method that accelerates this training process by 5× to 10×. By exploiting the unique mathematical properties of Gaussian kernels—specifically the "weak coupling" of their attributes—the authors replace slow, linear-converging gradient descent with a near second-order convergent algorithm. This allows for high-quality reconstructions in seconds rather than minutes.

The Challenge: Beyond First-Order Optimization

In deep learning and 3D reconstruction, SGD is the industry standard due to its simplicity and scalability. However, its linear convergence rate is inherently limited. Second-order methods, such as Newton’s method, offer quadratic convergence, which could theoretically reach the same loss levels in a fraction of the steps.

In the context of 3DGS, a standard Newton method is traditionally impossible. Calculating a "global Hessian" (a matrix of second derivatives) for millions of Gaussians is computationally ruinous in terms of both memory and processing power. 3DGS$^2$ overcomes this by asking a pivotal question: Can we achieve the speed of Newton's method without the impossible overhead of a global Hessian?

Methodology: Stochastic Local Newton

The authors introduce three primary innovations to make second-order optimization practical for 3DGS:

1. Parallelized Local Newton Solve

The researchers observed that the influence of a Gaussian kernel is localized; a change in one kernel's attributes rarely impacts all others globally. This "weak coupling" allows the system to ignore the global Hessian. Instead, the algorithm analytically derives and solves small, independent Newton systems for each kernel's attribute groups (position, scale, rotation, opacity, and color) in parallel across GPU threads.

2. Secondary Targets for Stability

Second-order methods are notoriously prone to "overshooting" in stochastic settings where only one training image is viewed at a time. To stabilize the optimization, 3DGS$^2$ identifies Secondary Targets—the $K$ nearest neighboring camera views to the current training image. By incorporating a downsampled loss from these neighbors, the optimizer gains a "spatial look-ahead." This prevents aggressive updates from improving one view while inadvertently ruining the reconstruction from nearby angles.

3. Prioritized Optimization Order

The method enforces a specific sequence for attribute updates: Position $\rightarrow$ Shape (Rotation/Scale) $\rightarrow$ Appearance (Opacity/Color). This hierarchy reflects the physical reality of the scene: if a Gaussian is not correctly positioned in 3D space, adjusting its color or opacity is computationally futile.

Key Insights

  • Hessian Sparsity: Visual analysis of the full Hessian in a 3DGS scene reveals that influence is highly localized. This "spiky" sparsity justifies the shift from global to local, per-kernel optimizations.
  • Eliminating Learning Rates: Unlike SGD, which requires meticulous tuning of learning rates to prevent divergence, a valid quadratic approximation (Newton's method) naturally utilizes a step size of 1.0. This significantly simplifies the hyperparameter search for new datasets.
  • Spatial Correlation as a Pre-conditioner: The use of neighboring views acts as a self-adjusting pre-conditioner, providing enough global context to keep the local Newton solver stable without the cost of processing the entire dataset at every iteration.

Experimental Results

The authors evaluated 3DGS$^2$ on standard benchmarks, including Mip-NeRF360, Tanks & Temples, and Deep Blender. The results demonstrate a paradigm shift in training efficiency:

  • Speed: The method is consistently 5× to 10× faster than vanilla 3DGS. A scene that requires over 6 minutes to train with standard methods can be completed in just 46 seconds using 3DGS$^2$.
  • Efficiency: The algorithm achieves superior or equal quality (measured by PSNR, SSIM, and LPIPS) using roughly one-tenth the iterations of gradient-based methods.
  • Visual Fidelity: In the "Deep Blender" dataset, 3DGS$^2$ achieved a PSNR of 30.43, outperforming vanilla 3DGS (29.90) while maintaining a significantly faster training profile.

Limitations and Future Work

While 3DGS$^2$ represents a major leap forward, it is currently optimized for static scenes. In dynamic environments, the coupling between Gaussians changes over time, which may require further refinements to the local Newton approach. Additionally, while the method is fast, evaluating the Hessian for the position attribute remains the most computationally intensive part of the pipeline. Future work may involve integrating spatial data structures like AABBs or spatial hashing to further alleviate bottlenecks in kernel sorting.

Summary updated

BibTeX
@inproceedings{lan2025dgs,
  author = {Lei Lan and Tianjia Shao and Zixuan Lu and Yu Zhang and Chenfanfu Jiang and Yin Yang},
  title = {{3DGS2: Near Second-order Converging 3D Gaussian Splatting}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730687},
}
Figure