Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting

Youngsik Yun, Jeongmin Bae, Hyun Seung Son, Seoha Kim, Hahyun Lee, Gun Bang, Youngjung Uh

Abstract

Online dynamic 3D reconstruction is essential for real-time applications such as VR/AR and telepresence. However, current methods often suffer from temporal instability and flickering because they overfit to per-frame sensor noise and compression artifacts. This paper introduces an "Observation-Restoring" framework that utilizes a learnable residual map to separate transient errors from the underlying scene. By providing a 2D outlet for high-frequency noise, the system produces smoother, more consistent dynamic reconstructions while simultaneously reducing the total number of Gaussian primitives required.

Introduction

Dynamic 3D reconstruction using 3D Gaussian Splatting (3DGS) has revolutionized the creation of digital twins from video sequences. While offline methods can average out noise by processing entire sequences simultaneously, "online" reconstruction must process frames as a live stream. This constraint makes online models highly susceptible to spatiotemporal inconsistency. Because these models only observe one frame at a time, they tend to overfit to subtle, high-frequency noise—such as sensor grain or compression artifacts—causing static regions of a scene to "shimmer" or flicker over time.

Methodology: Observation-Restoring Reconstruction

The core contribution of this work is the Observation-Restoring Online Reconstruction framework. The authors hypothesize that a captured video frame ($\tilde{I}_t$) is a composite of an "ideal" scene ($\hat{I}_t$) and a transient error component ($\hat{M}_t$).

1. Learnable Residual Maps

Rather than forcing the 3D Gaussians to match the noisy input exactly, the system jointly optimizes the 3D Gaussians and a 2D per-view residual map. This residual map is specifically designed to absorb high-frequency, view-inconsistent noise that does not belong in the 3D model. Mathematically, it is "easier" for a 2D residual map to fit per-pixel noise than it is for 3D Gaussians to adjust their spatial parameters (position, scale, and color), allowing the 3D model to naturally ignore the noise.

2. Gaussian Propagation

To maintain temporal continuity, the method propagates both deformed and newly added Gaussians to subsequent frames. This differs from previous online methods that often discarded new Gaussians after a single frame, ensuring that objects appearing in the scene for the first time maintain a consistent identity over time.

3. Sparse Regularization

To prevent the residual map from absorbing the entire image signal, the authors apply $L_1$ regularization. This encourages the residual map to remain sparse, capturing only the necessary corrections while leaving the actual scene structure to be represented by the 3D Gaussians.

Experimental Setup

The method was evaluated by integrating it into three state-of-the-art baselines: Dynamic3DG, 3DGStream, and HiCoM. Testing was conducted across several prominent benchmarks:

  • Neural 3D Video: Real-world scenes of human activities captured by 17–21 synchronized cameras.
  • MeetRoom: Multi-view captures of office environments.
  • Sync-NeRF: A synthetic dataset providing noise-free ground truth to validate restoration capabilities.
  • Modified "Fox" Scene: A synthetic dataset used for ablation studies regarding varying noise levels.

Key Results and Insights

Improved Visual Quality and Consistency

The integration of the proposed module improved PSNR (Peak Signal-to-Noise Ratio) by approximately 0.5 to 1.0 dB across most datasets. More importantly, temporal consistency—measured via masked Total Variation (mTV)—showed a reduction in flickering by up to 70% compared to the original baselines.

Efficiency and Robustness

By ignoring noise, the model becomes more efficient. The proposed method utilizes 20-30% fewer Gaussians than baseline methods because it avoids creating "phantom" geometry to account for sensor grain. This reduction in primitives leads to faster rendering speeds and a lower memory footprint. Additionally, the optimization process demonstrated lower variance across multiple runs, suggesting higher robustness for production environments.

Restoration Capabilities

In synthetic tests using the Sync-NeRF dataset, the method successfully "subtracted" artificial noise, recovering an image that was closer to the original ground truth than the noisy input itself. This confirms that the residual map effectively isolates errors rather than just blurring the output.

Limitations and Future Work

While the method significantly improves stability, it still faces challenges with extremely rapid motion or significant motion blur, where the "ideal" signal is already degraded at the source. Furthermore, as a "plug-in" enhancement, its performance is partially capped by the capabilities of the underlying reconstruction baseline. Future research may explore specialized residual modeling for different noise types, such as low-light Poisson noise or specific video compression artifacts.

Summary updated

BibTeX
@inproceedings{yun2025compensating,
  author = {Youngsik Yun and Jeongmin Bae and Hyun Seung Son and Seoha Kim and Hahyun Lee and Gun Bang and Youngjung Uh},
  title = {{Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730678},
}
Figure