Abstract
Dynamic 3D reconstruction of scenes with rapid, non-rigid motions—such as professional sports—remains a significant challenge for current 4D Gaussian Splatting (4DGS) techniques. Standard methods often struggle with "coupling," where a single canonical representation fails to map accurately to high-velocity movements, resulting in motion blur and "ghosting." RelayGS addresses this through a "divide-and-conquer" strategy. By introducing Relay Gaussians to decompose long trajectories and a learnable mask for foreground-background decoupling, RelayGS achieves state-of-the-art performance, outperforming existing baselines by up to 2 dB in PSNR while maintaining real-time rendering at 70 FPS.
Introduction
While 3D Gaussian Splatting (3DGS) has set new standards for static scene reconstruction, extending it to the temporal dimension (4DGS) introduces complexities. Most 4DGS frameworks attempt to deform a single "canonical" set of Gaussians across all frames using a neural motion field (typically MLPs). However, when objects move quickly or undergo complex non-rigid transformations (e.g., a basketball player's limbs during a jump), the MLP often fails to predict these large displacements. This leads to an optimization imbalance where the model overfits to static backgrounds while fast-moving foreground objects become blurred or disappear.
Methodology
RelayGS proposes a three-stage progressive pipeline designed to decouple static and dynamic elements and simplify the motion modeling task.
1. Foreground-Background Decoupling
Unlike traditional methods that rely on external segmentation models, RelayGS utilizes a learnable mask. During initial scene reconstruction, Gaussians associated with high-dynamic regions naturally accumulate higher errors. The learnable mask identifies these points, effectively separating the foreground (e.g., players) from the background (e.g., the court) without requiring manual labels or pre-trained optical flow.
2. Trajectory Decomposition via Relay Gaussians
The core innovation of RelayGS is the use of Relay Gaussians. The foreground Gaussians are replicated at specific temporal intervals (e.g., every 16 frames). These replicas serve as physical "checkpoints" or "relays" along the motion path. To optimize these relays, the authors employ pseudo-views—composite images blended from multiple frames—which provide a denser supervisory signal, helping the Gaussians settle into accurate positions across time segments.
3. Spatiotemporal Modeling
In the final stage, the model jointly optimizes the canonical Gaussians and a compact motion field based on the HexPlane architecture. Crucially, separate MLPs are used for the background and the Relay Gaussians. A learnable scaling factor ($\gamma$) is added to position offsets, enabling the model to capture significantly larger physical displacements than standard MLPs can typically handle.
Key Innovations
- Temporal Densification: Similar to how 3DGS adds points in space to improve detail, RelayGS adds "points in time" via Relay Gaussians to bridge gaps in fast motion.
- MLP Task Simplification: By providing the motion field with "Relay" starting points already near their targets, the MLPs no longer need to "teleport" Gaussians across large distances, leading to more stable optimization.
- Adaptive Masking: The learnable mask allows the model to autonomously decide which scene elements require the complex "Relay" pipeline versus simple static modeling.
Experimental Results
The researchers evaluated RelayGS on the PanopticSports and VRU Basketball Games datasets. The latter is particularly challenging, featuring 1080p resolution and 34 synchronized cameras capturing intense basketball action.
- Quantitative Performance: RelayGS achieved a 1.02 dB PSNR improvement on average across PanopticSports and a 2 dB improvement on the VRU Basketball dataset compared to baseline 4D-GS.
- Qualitative Clarity: Visual comparisons demonstrate that RelayGS is uniquely capable of reconstructing fast-moving objects like basketballs and limbs without the transparency or geometry loss seen in competing methods.
- Efficiency: The system maintains a rendering speed of approximately 70 FPS on an NVIDIA RTX 4090, proving its suitability for real-time applications.
Conclusion and Future Work
RelayGS represents a significant step forward in dynamic scene reconstruction by effectively handling large-scale and complex motions. While highly effective, the authors note that extremely small, fast objects still pose tracking challenges. Future work may involve adaptive temporal segmentation, where the model dynamically adjusts the frequency of Relay Gaussians based on the speed of the action, further optimizing the balance between reconstruction quality and computational overhead.
Summary updated