RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos

Yoonwoo Jeong, Junmyeong Lee, Hoseung Choi, Minsu Cho

TL;DR

This paper introduces RoDyGS, a novel method for reconstructing dynamic 3D scenes from casual, monocular videos. It achieves this by separating static and dynamic scene elements and applying novel regularization techniques to ensure physically plausible geometry and motion. The authors also introduce Kubric-MRig, a new benchmark dataset designed to better evaluate dynamic scene reconstruction methods.

What Problem Does This Paper Address?

Capturing dynamic real-world scenes with everyday devices like smartphones often results in monocular videos. These videos lack crucial 3D information, such as precise camera poses and scene geometry, making it challenging to reconstruct accurate dynamic 3D representations. Existing methods often struggle with the inherent ambiguity in these casual videos, leading to artifacts, inconsistent geometry, and poor motion representation, especially when dealing with occlusions and complex object movements.

Methods

RoDyGS tackles this challenge with a multi-pronged approach:

  • Dynamic Gaussian Splatting (DGS): Building upon the efficient 3D Gaussian Splatting (3DGS) framework, RoDyGS extends it to handle dynamic scenes.
  • Static-Dynamic Part Separation: A key innovation is the explicit separation of scene elements into static and dynamic Gaussian sets. This allows for tailored optimization strategies: static elements are optimized with camera poses, while dynamic elements benefit from specific motion and geometric regularizations. This disentanglement helps reduce camera-object motion ambiguity.
  • Spatiotemporal Regularization: To ensure physically plausible geometry and temporally consistent motion for dynamic elements, RoDyGS employs:
  • Distance-Preserving Regularization: Enforces that the distances between dynamic Gaussians remain consistent over time, even during occlusions.
  • Object Surface Smoothing: Promotes local surface smoothness for dynamic objects.
  • Steady Motion Regularization: Leverages the observation that object motion can often be approximated by low-rank 3D motion, using learnable motion bases and a continuity constraint.
  • Weighted Motion Regularization: A refinement of steady motion regularization that adaptively applies weights to motion basis elements, preventing over-smoothing and preserving fine details.
  • Camera Parameter Refinement: RoDyGS uses off-the-shelf priors (like MASt3R for initial poses and DepthAnything for depth) to initialize camera parameters and scene geometry, then refines them during optimization.

Datasets

  • Kubric-MRig: A newly introduced benchmark dataset designed to address limitations in existing datasets. It features photorealistic scenes with extensive camera and object motion, simultaneous multi-view captures, and ground truth camera poses and background. It's generated using Blender.
  • iPhone Dataset [15]: Used for evaluation, this dataset is adapted to be more suitable for pose-free dynamic novel view synthesis by ensuring test cameras are close to training cameras.
  • Tanks and Temples Dataset [30]: Used for evaluating static novel view synthesis capabilities.
  • NVIDIA Dynamic Dataset [71]: Used to evaluate performance on scenes with ambient camera motion.

Key Insights and Ideas

  • Part Separation is Crucial: Explicitly separating static and dynamic scene elements is a powerful technique for reducing ambiguity and enabling more stable and effective optimization in dynamic scenes.
  • Regularization is Key for Plausibility: The proposed spatiotemporal and motion regularizations are essential for achieving physically plausible geometry and natural-looking motion, especially in challenging scenarios with occlusions.
  • A Better Benchmark Matters: The creation of Kubric-MRig highlights the need for datasets that better reflect real-world casual video capture conditions to truly push the boundaries of dynamic scene reconstruction.
  • Pose-Free is Achievable and Competitive: RoDyGS demonstrates that high-quality dynamic scene reconstruction can be achieved from casual videos without requiring pre-calibrated camera poses, outperforming previous pose-free methods.

Key Results

  • Superior Performance on Kubric-MRig: RoDyGS significantly outperforms prior pose-free dynamic novel view synthesis methods on the Kubric-MRig dataset.
  • Competitive on iPhone: Achieves competitive results on the iPhone dataset, even outperforming methods that use ground-truth camera poses.
  • Comparable to Static NVS: RoDyGS demonstrates rendering quality comparable to state-of-the-art pose-free static novel view synthesis approaches on the Tanks and Temples dataset.
  • Qualitative Improvements: Visualizations show RoDyGS produces sharper renderings with better geometric accuracy and temporal consistency, with fewer artifacts like ghosting or shape distortions compared to baselines.

Limitations and Future Work

  • Reconstruction of Observed Parts Only: Like many 4D reconstruction methods, RoDyGS can only reconstruct parts of the scene that are visible in the input videos. Completely occluded or long-term occluded parts are not recovered. The authors suggest leveraging generative priors (e.g., diffusion models) to address this.
  • Limited to Non-Rigid Deformations: The current regularization implicitly assumes dynamic scene elements do not exhibit highly complex, non-rigid deformations (e.g., human bodies, loose clothing). Future work could incorporate parametric models (like SMPL) or other data-driven priors to handle such cases.
  • Automating Dynamic Part Separation: The current method relies on some level of user interaction for separating static and dynamic parts. Automating this process is a key future direction.
  • Improving Camera Calibration Robustness: RoDyGS's performance is dependent on initial camera pose estimates. Future research aims to develop more robust joint camera optimization techniques to handle inaccurate initializations.

Summary updated

Abstract

Dynamic view synthesis (DVS) has advanced remarkably in recent years, achieving high-fidelity rendering while reducing computational costs. Despite the progress, optimizing dynamic neural fields from casual videos remains challenging, as these videos do not provide direct 3D information, such as camera trajectories or the underlying scene geometry. In this work, we present RoDyGS, an optimization pipeline for dynamic Gaussian Splatting from casual videos. It effectively learns motion and underlying geometry of scenes by separating dynamic and static primitives, and ensures that the learned motion and geometry are physically plausible by incorporating motion and geometric regularization terms. We also introduce a comprehensive benchmark, Kubric-MRig, that provides extensive camera and object motion along with simultaneous multi-view captures, features that are absent in previous benchmarks. Experimental results demonstrate that the proposed method significantly outperforms previous pose-free dynamic neural fields and achieves competitive rendering quality compared to existing pose-free static neural fields. The code and data are publicly available at https://rodygs.github.io/.
BibTeX
@misc{jeong2024rodygs,
  author = {Yoonwoo Jeong and Junmyeong Lee and Hoseung Choi and Minsu Cho},
  title = {{RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos}},
  year = {2024},
  eprint = {2412.03077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.03077},
}
Figure