SWAGS: Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting

Richard Shaw, Jifei Song, Arthur Moreau, Michal Nazarczuk, Sibi Catley-Chandar, Helisa Dhamo, Eduardo Pérez-Pellitero

TL;DR

This paper introduces SWinGS, a novel method that extends the popular 3D Gaussian Splatting (3DGS) technique to handle dynamic scenes. By employing a sliding window approach with adaptive sampling and dynamic MLPs, SWinGS achieves high-quality, real-time rendering of complex dynamic environments, overcoming limitations of previous static-scene methods.

What Problem Does This Paper Address?

Recent advancements in novel view synthesis have seen 3D Gaussian Splatting (3DGS) emerge as a powerful technique for generating photorealistic renderings and enabling real-time interactive viewing. However, a significant limitation of 3DGS is its applicability only to static scenes. Reconstructing dynamic scenes, where objects and environments change over time, presents a substantial challenge. Existing methods often struggle with long sequences, suffer from temporal inconsistencies (flickering), or require computationally expensive deformation fields. This paper aims to bridge this gap by extending 3DGS to effectively model and render dynamic scenes with high fidelity and temporal coherence.

Methods

SWinGS builds upon the foundation of 3DGS but introduces several key innovations to handle dynamic scenes:

  • Dynamic 3D Gaussians: Instead of static Gaussians, SWinGS uses dynamic MLPs to learn deformation fields. These MLPs map from a temporally-local canonical representation to per-frame 3D Gaussians, allowing them to move and change over time.
  • Sliding Window Training: To manage arbitrary-length sequences and prevent degradation in quality, the input sequence is partitioned into smaller, overlapping sliding windows. This allows for independent training of dynamic 3DGS models for each window, enabling the canonical representation to adapt to significant geometric changes within that window.
  • Adaptive Window Sampling: The length of these sliding windows is not fixed but is adaptively determined based on the amount of motion detected in the scene (using optical flow). This strategy balances training overhead with the need to capture scene dynamics effectively, sampling more frequently in high-motion areas and less frequently in low-motion areas.
  • Tunable Dynamic MLPs: To address scenes with imbalanced static and dynamic regions, SWinGS employs tunable parameters within the MLPs. These parameters weigh the MLP's contributions for each Gaussian, allowing the model to disentangle static and dynamic components and learn different "modes" of motion.
  • Temporal Consistency Fine-tuning: After training individual window models, a fine-tuning step is applied to enforce temporal consistency across window boundaries. This involves a self-supervised loss function that encourages smooth transitions in novel views rendered from overlapping frames between adjacent windows, mitigating flickering.

Datasets

The paper evaluates SWinGS on two real-world multi-view dynamic benchmarks:

  • Neural 3D Video dataset [25]: This dataset features dynamic scenes captured with a time-synchronized multi-view system.
  • Technicolor dataset [51]: This dataset captures real dynamic scenes using a synchronized 4x4 camera array.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Windowing is Crucial for Dynamics: The core insight is that treating dynamic scenes as a series of locally consistent, but independently evolving, temporal segments (windows) is more effective than trying to model the entire sequence with a single, global representation. This allows for adaptation to significant geometric changes.
  • Adaptive Windowing Balances Performance: Dynamically adjusting window sizes based on scene motion is key to achieving a good trade-off between computational cost, memory usage, and the ability to capture complex dynamics without excessive overhead.
  • Disentangling Static and Dynamic Regions is Essential: For scenes with varying levels of motion, explicitly modeling the distinction between static and dynamic elements (via tunable MLPs) is vital for accurate deformation estimation and preventing artifacts.
  • Fine-tuning for Temporal Coherence is a Must: Even with windowing, ensuring smooth transitions between these independently trained segments requires a dedicated temporal consistency step, which significantly improves the perceptual quality of the final output.

Key Results

  • State-of-the-Art Performance: SWinGS achieves competitive quantitative results, often outperforming existing methods in metrics like PSNR and SSIM on the Neural 3D Video dataset.
  • Real-time Rendering: The method enables real-time interactive viewing, a significant advantage for dynamic scene exploration.
  • Improved Temporal Consistency: Ablation studies demonstrate a substantial improvement in temporal consistency (measured by t-LPIPS and FAST-VQA) after applying the temporal fine-tuning step, leading to more pleasing renderings with reduced flickering.
  • Effective Handling of Complex Motion: Qualitative results showcase SWinGS's ability to render complex dynamic phenomena like flames with high fidelity.

Limitations and Future Work

  • Per-frame Metric Trade-off: While temporal consistency is significantly improved, there's a minor penalty in some per-frame performance metrics (SSIM and LPIPS) compared to methods that might not handle dynamics as well.
  • Adaptive Windowing Complexity: While beneficial, the adaptive window sampling strategy adds a layer of complexity to the training pipeline.
  • Future Work: The paper suggests that the adaptive window sampling strategy could be further refined. Additionally, the method's ability to handle complex topological changes and its potential for real-time editing of dynamic scenes are areas for future exploration.

Summary updated

Abstract

Novel view synthesis has shown rapid progress recently, with methods capable of producing increasingly photorealistic results. 3D Gaussian Splatting has emerged as a promising method, producing high-quality renderings of scenes and enabling interactive viewing at real-time frame rates. However, it is limited to static scenes. In this work, we extend 3D Gaussian Splatting to reconstruct dynamic scenes. We model a scene's dynamics using dynamic MLPs, learning deformations from temporally-local canonical representations to per-frame 3D Gaussians. To disentangle static and dynamic regions, tuneable parameters weigh each Gaussian's respective MLP parameters, improving the dynamics modelling of imbalanced scenes. We introduce a sliding window training strategy that partitions the sequence into smaller manageable windows to handle arbitrary length scenes while maintaining high rendering quality. We propose an adaptive sampling strategy to determine appropriate window size hyperparameters based on the scene's motion, balancing training overhead with visual quality. Training a separate dynamic 3D Gaussian model for each sliding window allows the canonical representation to change, enabling the reconstruction of scenes with significant geometric changes. Temporal consistency is enforced using a fine-tuning step with self-supervising consistency loss on randomly sampled novel views. As a result, our method produces high-quality renderings of general dynamic scenes with competitive quantitative performance, which can be viewed in real-time in our dynamic interactive viewer.
BibTeX
@inproceedings{shaw2024swags,
  author = {Richard Shaw and Jifei Song and Arthur Moreau and Michal Nazarczuk and Sibi Catley-Chandar and Helisa Dhamo and Eduardo Pérez-Pellitero},
  title = {{SWAGS: Sampling Windows Adaptively for Dynamic 3D Gaussian Splatting}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73001-6\_3},
}
Figure