RelayGS: Reconstructing Dynamic Scenes with Large-Scale and Complex Motions via Relay Gaussians

Qiankun Gao, Yanmin Wu, Chengxiang Wen, Jiarui Meng, Luyang Tang, Jie Chen, Ronggang Wang, Jian ZhangarXiv 2024 · DOI · arXiv

Abstract

Dynamic 3D reconstruction of scenes with rapid, non-rigid motions—such as professional sports—remains a significant challenge for current 4D Gaussian Splatting (4DGS) techniques. Standard methods often struggle with "coupling," where a single canonical representation fails to map accurately to high-velocity movements, resulting in motion blur and "ghosting." RelayGS addresses this through a "divide-and-conquer" strategy. By introducing Relay Gaussians to decompose long trajectories and a learnable mask for foreground-background decoupling, RelayGS achieves state-of-the-art performance, outperforming existing baselines by up to 2 dB in PSNR while maintaining real-time rendering at 70 FPS.

Introduction

While 3D Gaussian Splatting (3DGS) has set new standards for static scene reconstruction, extending it to the temporal dimension (4DGS) introduces complexities. Most 4DGS frameworks attempt to deform a single "canonical" set of Gaussians across all frames using a neural motion field (typically MLPs). However, when objects move quickly or undergo complex non-rigid transformations (e.g., a basketball player's limbs during a jump), the MLP often fails to predict these large displacements. This leads to an optimization imbalance where the model overfits to static backgrounds while fast-moving foreground objects become blurred or disappear.

Methodology

RelayGS proposes a three-stage progressive pipeline designed to decouple static and dynamic elements and simplify the motion modeling task.

The three-stage pipeline of RelayGS: Scene Initialization, Trajectory Decomposition, and Spatiotemporal Modeling.

1. Foreground-Background Decoupling

Unlike traditional methods that rely on external segmentation models, RelayGS utilizes a learnable mask. During initial scene reconstruction, Gaussians associated with high-dynamic regions naturally accumulate higher errors. The learnable mask identifies these points, effectively separating the foreground (e.g., players) from the background (e.g., the court) without requiring manual labels or pre-trained optical flow.

2. Trajectory Decomposition via Relay Gaussians

The core innovation of RelayGS is the use of Relay Gaussians. The foreground Gaussians are replicated at specific temporal intervals (e.g., every 16 frames). These replicas serve as physical "checkpoints" or "relays" along the motion path. To optimize these relays, the authors employ pseudo-views—composite images blended from multiple frames—which provide a denser supervisory signal, helping the Gaussians settle into accurate positions across time segments.

3. Spatiotemporal Modeling

In the final stage, the model jointly optimizes the canonical Gaussians and a compact motion field based on the HexPlane architecture. Crucially, separate MLPs are used for the background and the Relay Gaussians. A learnable scaling factor ($\gamma$) is added to position offsets, enabling the model to capture significantly larger physical displacements than standard MLPs can typically handle.

Key Innovations

  • Temporal Densification: Similar to how 3DGS adds points in space to improve detail, RelayGS adds "points in time" via Relay Gaussians to bridge gaps in fast motion.
  • MLP Task Simplification: By providing the motion field with "Relay" starting points already near their targets, the MLPs no longer need to "teleport" Gaussians across large distances, leading to more stable optimization.
  • Adaptive Masking: The learnable mask allows the model to autonomously decide which scene elements require the complex "Relay" pipeline versus simple static modeling.

Experimental Results

The researchers evaluated RelayGS on the PanopticSports and VRU Basketball Games datasets. The latter is particularly challenging, featuring 1080p resolution and 34 synchronized cameras capturing intense basketball action.

  • Quantitative Performance: RelayGS achieved a 1.02 dB PSNR improvement on average across PanopticSports and a 2 dB improvement on the VRU Basketball dataset compared to baseline 4D-GS.
  • Qualitative Clarity: Visual comparisons demonstrate that RelayGS is uniquely capable of reconstructing fast-moving objects like basketballs and limbs without the transparency or geometry loss seen in competing methods.
  • Efficiency: The system maintains a rendering speed of approximately 70 FPS on an NVIDIA RTX 4090, proving its suitability for real-time applications.

Conclusion and Future Work

RelayGS represents a significant step forward in dynamic scene reconstruction by effectively handling large-scale and complex motions. While highly effective, the authors note that extremely small, fast objects still pose tracking challenges. Future work may involve adaptive temporal segmentation, where the model dynamically adjusts the frequency of Relay Gaussians based on the speed of the action, further optimizing the balance between reconstruction quality and computational overhead.

Summary updated

Abstract

Reconstructing dynamic scenes with large-scale and complex motions remains a significant challenge. Recent techniques like Neural Radiance Fields and 3D Gaussian Splatting (3DGS) have shown promise but still struggle with scenes involving substantial movement. This paper proposes RelayGS, a novel method based on 3DGS, specifically designed to represent and reconstruct highly dynamic scenes. Our RelayGS learns a complete 4D representation with canonical 3D Gaussians and a compact motion field, consisting of three stages. First, we learn a fundamental 3DGS from all frames, ignoring temporal scene variations, and use a learnable mask to separate the highly dynamic foreground from the minimally moving background. Second, we replicate multiple copies of the decoupled foreground Gaussians from the first stage, each corresponding to a temporal segment, and optimize them using pseudo-views constructed from multiple frames within each segment. These Gaussians, termed Relay Gaussians, act as explicit relay nodes, simplifying and breaking down large-scale motion trajectories into smaller, manageable segments. Finally, we jointly learn the scene's temporal motion and refine the canonical Gaussians learned from the first two stages. We conduct thorough experiments on two dynamic scene datasets featuring large and complex motions, where our RelayGS outperforms state-of-the-arts by more than 1 dB in PSNR, and successfully reconstructs real-world basketball game scenes in a much more complete and coherent manner, whereas previous methods usually struggle to capture the complex motion of players. Code will be publicly available at https://github.com/gqk/RelayGS
BibTeX
@misc{gao2024relaygs,
  author = {Qiankun Gao and Yanmin Wu and Chengxiang Wen and Jiarui Meng and Luyang Tang and Jie Chen and Ronggang Wang and Jian Zhang},
  title = {{RelayGS: Reconstructing Dynamic Scenes with Large-Scale and Complex Motions via Relay Gaussians}},
  year = {2024},
  eprint = {2412.02493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.02493},
}
Figure