4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time Rendering of Temporally Complex Dynamic Scenes

Jinbo Yan, Rui Peng, Luyang Tang, Ronggang Wang

Abstract

Reconstructing dynamic scenes from video sequences is a highly promising task in the multimedia domain. While previous methods have made progress, they often struggle with slow rendering and managing temporal complexities such as significant motion and object appearance/disappearance. In this paper, we propose SaRO-GS as a novel dynamic scene representation capable of achieving real-time rendering while effectively handling temporal complexities in dynamic scenes. To address the issue of slow rendering speed, we adopt a Gaussian primitive-based representation and optimize the Gaussians in 4D space, which facilitates real-time rendering with the assistance of 3D Gaussian Splatting. Additionally, to handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field. This field considers the size information of each Gaussian primitive while encoding its residual feature and aligns with the self-splitting behavior of Gaussian primitives. Furthermore, we propose an Adaptive Optimization Schedule, which assigns different optimization strategies to Gaussian primitives based on their distinct temporal properties, thereby expediting the reconstruction of dynamic regions. Through evaluations on monocular and multi-view datasets, our method has demonstrated state-of-the-art performance.

Introduction

Reconstructing dynamic 3D scenes from video sequences is a critical challenge for immersive technologies like VR, AR, and the metaverse. While Neural Radiance Fields (NeRF) produce high-fidelity results, their computational intensity leads to slow rendering speeds. 3D Gaussian Splatting (3DGS) revolutionized static scene reconstruction by enabling real-time rendering through point-based rasterization. However, extending 3DGS to dynamic (4D) environments introduces two primary difficulties: managing temporal complexity—such as fast motion and transient objects—and maintaining field consistency during Gaussian densification (splitting).

Methodology

SaRO-GS represents dynamic scenes using 4D Gaussian primitives optimized alongside a specialized feature field to ensure temporal and spatial coherence.

4D Gaussian Primitives with Lifespans

Unlike static Gaussians, each primitive in SaRO-GS is assigned a temporal position ($\tau$) and a lifespan ($\sigma$). A Gaussian-like state function determines the primitive's opacity at any time $t$. This explicit modeling allows the system to accurately represent objects that appear or disappear within a sequence, effectively managing the "transient" nature of dynamic scenes.

Scale-aware Residual Field

To address inconsistencies during Gaussian splitting, SaRO-GS utilizes a Scale-aware Residual Field. Standard grid-based fields (like Hex-planes) can cause artifacts when a Gaussian splits into smaller children that query the same point-based feature. By incorporating a MipMap stack, SaRO-GS allows Gaussians to query the field based on their scale: larger Gaussians query lower-resolution levels, while smaller ones query higher-resolution levels, ensuring feature continuity throughout the optimization process.

The SaRO-GS pipeline: 4D Gaussians are projected to 3D space for splatting, with residuals queried from a Scale-aware Residual Field.

Adaptive Optimization Schedule

Dynamic scenes often suffer from "gradient imbalance," where static regions are over-optimized compared to transient ones. SaRO-GS introduces an Adaptive Optimization Schedule that calculates a temporal integral for each Gaussian. By dynamically boosting learning rates and lowering densification thresholds for Gaussians with shorter lifespans, the framework ensures that fast-moving foreground elements are reconstructed with the same clarity as the background.

Key Results

SaRO-GS sets a new benchmark for both speed and quality in dynamic scene reconstruction. On the D-NeRF synthetic dataset, it achieves a PSNR of 36.13 dB and a rendering speed of 182 FPS, representing up to an 80x speedup over NeRF-based alternatives like HexPlane. On high-resolution real-world datasets, it maintains a smooth 40-50 FPS.

Performance comparison showing FPS vs. PSNR. SaRO-GS (Ours) significantly outperforms existing methods like 4DGS and HexPlane in both efficiency and reconstruction quality.

Conclusion

SaRO-GS provides a high-performance framework for real-time 4D rendering. By aligning the feature field with the physical scale of Gaussian primitives and balancing optimization across time, it effectively handles complex dynamic scenes. Future research will explore its application in 4D content generation, robust object tracking, and interactive scene editing.

Summary updated

Abstract

Reconstructing dynamic scenes from video sequences is a highly promising task in the multimedia domain. While previous methods have made progress, they often struggle with slow rendering and managing temporal complexities such as significant motion and object appearance/disappearance. In this paper, we propose SaRO-GS as a novel dynamic scene representation capable of achieving real-time rendering while effectively handling temporal complexities in dynamic scenes. To address the issue of slow rendering speed, we adopt a Gaussian primitive-based representation and optimize the Gaussians in 4D space, which facilitates real-time rendering with the assistance of 3D Gaussian Splatting. Additionally, to handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field. This field considers the size information of each Gaussian primitive while encoding its residual feature and aligns with the self-splitting behavior of Gaussian primitives. Furthermore, we propose an Adaptive Optimization Schedule, which assigns different optimization strategies to Gaussian primitives based on their distinct temporal properties, thereby expediting the reconstruction of dynamic regions. Through evaluations on monocular and multi-view datasets, our method has demonstrated state-of-the-art performance. Please see our project page at https://yjb6.github.io/SaRO-GS.github.io.
BibTeX
@inproceedings{yan2024d,
  author = {Jinbo Yan and Rui Peng and Luyang Tang and Ronggang Wang},
  title = {{4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time Rendering of Temporally Complex Dynamic Scenes}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=bgRADWSr6O},
}
Figure