Fast and Robust Deformable 3D Gaussian Splatting

Han Jiao, Jiakai Sun, Liguo Zhao, Zhanjie Zhang, Wei Xing, Huaizhong Lin

TL;DR

This paper introduces FRoG, a novel framework for fast and robust dynamic 3D scene reconstruction using deformable 3D Gaussians. FRoG significantly speeds up rendering by employing an "early fusion" of temporal embeddings and improves robustness in sparse regions and dim scenes through a new sampling strategy and aggressive opacity reduction.

What Problem Does This Paper Address?

Reconstructing dynamic 3D scenes from video is a crucial task for applications like VR/AR. While recent methods like 3D Gaussian Splatting (3DGS) offer impressive rendering quality and speed for static scenes, extending them to dynamic scenes presents several challenges. Existing deformation-based methods, which warp a canonical scene representation over time, often suffer from slow rendering speeds due to complex fusion strategies, poor handling of sparse initial data (leading to artifacts), and a tendency to produce color shifts in dim or shadowed areas. FRoG aims to address these limitations by creating a more efficient and robust dynamic 3DGS system.

Methods

FRoG introduces several key innovations:

  • Early Temporal Embedding Fusion: Instead of the "late fusion" approach used in prior work (like E-D3DGS), which applies deformation multiple times, FRoG fuses "coarse" and "fine" temporal embeddings early using a Hadamard product. This significantly reduces computational overhead during rendering while maintaining quality.
  • Canonical Field Sampling Strategy: To combat issues arising from sparse initial point clouds, FRoG employs a novel sampling strategy. It identifies regions with high rendering errors, uses median depth to accurately estimate surfaces, and then injects new 3D Gaussians as "low-cost anchors" directly into the canonical field. This simplifies the deformation field's learning task, preventing it from having to learn complex, long-range transformations to fill in missing geometry.
  • Aggressive Opacity Reduction: To mitigate local optima problems in dim scenes (where "shadow floaters" can cause color bias), FRoG implements an enhanced opacity attenuation function. This strategy dynamically reduces the opacity of problematic "shadow floaters" during deformation, leading to improved color fidelity.

Datasets

FRoG was evaluated on three datasets:

  • N3DV: Features scenes with multiple objects and long video sequences, captured by 20 fixed cameras.
  • HyperNeRF: Includes videos captured by two fixed smartphones across various scenes.
  • Technicolor: Utilizes videos from 16 fixed cameras, known for its relatively simple motions but complex details.

The paper also references results from other methods on datasets like flame_salmon and cut_roasted_beef for comparative analysis.

Key Insights and Ideas

  • Early Fusion is Key for Speed: The paper demonstrates that fusing temporal information early, rather than late, is a critical optimization for achieving fast rendering in dynamic 3DGS.
  • Canonical Field Stability is Crucial: The success of the sampling strategy relies on the observation that the learned canonical field is a stable and coherent representation of the scene, making it a reliable base for injecting new geometry.
  • Median Depth for Robustness: Using median depth for identifying problematic regions is shown to be more robust than mean depth, which is susceptible to "floater" artifacts.
  • Injecting Anchors, Not Just Deforming: Directly injecting new Gaussians as anchors in sparse regions is a more effective and stable approach than forcing the deformation field to learn complex inverse mappings.
  • Opacity as a Dynamic Error Corrector: Opacity manipulation can be used not just for regularization but as an active mechanism to correct for dynamic scene artifacts like shadow floaters.

Key Results

FRoG achieves a compelling balance of speed and quality.

  • Rendering Speed: FRoG delivers fast rendering speeds, with FPS figures often exceeding those of comparable methods (e.g., 91.13 FPS on N3DV, 146.50 FPS on HyperNeRF).
  • Rendering Quality: It achieves state-of-the-art or competitive results in terms of PSNR, SSIM, and LPIPS across the tested datasets, often outperforming baselines in challenging dynamic regions and dim scenes. For instance, on N3DV, it achieved a PSNR of 31.01 and SSIM of 0.941.
  • Efficiency: FRoG boasts reduced storage requirements and competitive training times compared to many advanced methods.
  • Robustness: Qualitative results show FRoG's superior ability to reconstruct details in dynamic areas and its improved color fidelity in dim scenes compared to methods like E-D3DGS and 4DGaussians.

Limitations and Future Work

  • Monocular Scene Challenges: The paper acknowledges that FRoG, like many other methods, struggles with scenes that have significant missing view information, particularly in monocular videos. This can lead to blurry reconstructions and artifacts in regions with severe occlusion.
  • Future Work: The authors suggest that introducing strong external priors, such as using pre-trained 2D diffusion models to infer and generate plausible completions for occluded regions, could be a promising direction to address the limitations in monocular scenarios.

Summary updated

Abstract

3D Gaussian Splatting has demonstrated remarkable real-time rendering capabilities and superior visual quality in novel view synthesis for static scenes. Building upon these advantages, researchers have progressively extended 3D Gaussians to dynamic scene reconstruction. Deformation field-based methods have emerged as a promising approach among various techniques. These methods maintain 3D Gaussian attributes in a canonical field and employ the deformation field to transform this field across temporal sequences. Nevertheless, these approaches frequently encounter challenges such as suboptimal rendering speeds, significant dependence on initial point clouds, and vulnerability to local optima in dim scenes. To overcome these limitations, we present FRoG, an efficient and robust framework for high-quality dynamic scene reconstruction. FRoG integrates per-Gaussian embedding with a coarse-to-fine temporal embedding strategy, accelerating rendering through the early fusion of temporal embeddings. Moreover, to enhance robustness against sparse initializations, we introduce a novel depth- and error-guided sampling strategy. This strategy populates the canonical field with new 3D Gaussians at low-deviation initial positions, significantly reducing the optimization burden on the deformation field and improving detail reconstruction in both static and dynamic regions. Furthermore, by modulating opacity variations, we mitigate the local optima problem in dim scenes, improving color fidelity. Comprehensive experimental results validate that our method achieves accelerated rendering speeds while maintaining state-of-the-art visual quality.
BibTeX
@article{jiao2026fast,
  author = {Han Jiao and Jiakai Sun and Liguo Zhao and Zhanjie Zhang and Wei Xing and Huaizhong Lin},
  title = {{Fast and Robust Deformable 3D Gaussian Splatting}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2026},
  doi = {10.1109/tvcg.2026.3675272},
}
Figure