TL;DR
ATGS (Anchored Temporal Gaussian Splatting) introduces a novel framework for reconstructing high-quality volumetric videos of long sequences with complex, fast-moving actions. By organizing 3D Gaussians around time-conditioned anchors and employing a temporal windowing strategy, it successfully scales volumetric video representation to thousands of frames in a single training run without suffering from motion drift or severe visual artifacts.
What Problem Does This Paper Address?
Volumetric video enables immersive, free-viewpoint rendering of dynamic real-world scenes, which is crucial for applications like VR/AR, gaming, and sports broadcasting. While recent advancements in 3D Gaussian Splatting (3DGS) have made real-time, high-fidelity static rendering possible, extending this to the temporal domain remains a massive challenge.
Existing dynamic neural rendering methods face a double whammy: long-sequence reconstruction (minute-level videos) and complex motion modeling (fast, multi-person movements). Deformation-based approaches try to warp a canonical space over time, but they struggle to track long-term trajectories and suffer from accumulated optimization errors. Conversely, state-of-the-art 4D Gaussian methods are often constrained to short clips of just a few seconds (20 to 300 frames) and fail to scale, leading to temporal instability, blurriness, and missing geometry over extended durations.
Methods
Instead of explicitly tracking the long-term trajectories of individual Gaussian primitives—an inherently unstable process—ATGS decomposes long-term dynamics into a collection of temporally localized modeling tasks using three core components:
- Time-Conditioned Anchors: The scene geometry is represented by a sparse set of anchors initialized from periodically sampled keyframes. Each anchor is bound to a specific spatial position, scale, and discrete temporal index. Instead of moving across space, these anchors act as local priors that control the appearance and disappearance of local Gaussian primitives.
- Temporal Windowing Strategy: To keep temporal complexity under control during training and inference, the framework uses a sliding temporal window. Only anchors relevant to the queried time step are activated, preventing interference from distant frames and ensuring smooth temporal transitions.
- Hierarchical Spatio-Temporal Features: To stabilize Gaussian generation, each anchor aggregates three levels of features:
- Anchor features ($f_a$): Independent, learnable features for fine-grained local control.
- Static spatial features ($f_s$): Queried from a shared spatial grid to enforce local spatial consistency and prevent temporal jitter.
- Local temporal features ($f_t$): Queried from a partitioned set of 4D hash grids to capture fast-changing dynamics within localized time intervals.
A lightweight MLP decoder takes these fused features and generates the final parameters (position, rotation, scale, opacity, color) of the Temporal Gaussians used for volumetric rendering.
Datasets
The authors evaluated ATGS on several challenging multi-view, long-sequence benchmarks:
- N3DV (Neural 3D Video Dataset): Includes scenes with up to 1200 frames captured from 19–21 cameras.
- VRU (VRU-basketball): A highly challenging benchmark featuring large-scale, fast athletic motions across 1400 frames, captured by 36 cameras.
- MeetRoom: A sparse-view dataset consisting of forward-facing scenes captured by 11–12 cameras.
- SelfCap & PKU-DyMVHumans: Large-scale datasets containing complex human motions over thousands of frames used to validate generalization capability.
Key Insights and Ideas
- Anchors over Trajectories: Explicitly tracking fast-moving points across thousands of frames is a recipe for optimization failure. Treating motion as the localized appearance/disappearance of Gaussians anchored in space-time simplifies the optimization landscape.
- Feature Disentanglement Matters: Merely adding temporal features to an anchor can cause static and dynamic elements to blend, leading to a 10% training slowdown and blurry results. Explicitly separating the features into a shared static spatial grid and localized temporal hash grids allows the model to decouple static background geometry from rapid foreground motion.
Key Results
ATGS demonstrates significant quantitative and qualitative improvements over prior state-of-the-art methods like 4DGS and LocalDyGS:
- Long-Sequence Capability: ATGS successfully reconstructs a 1400-frame sequence (about 1 minute of video) in a single training run, representing a 70× increase in temporal coverage over methods like LocalDyGS, which typically break down after 20 frames of fast motion.
- Rendering Quality: On the VRU Long dataset, ATGS achieves a PSNR of 24.78 dB, outperforming LocalDyGS (23.21 dB) and 4DGS (23.01 dB). On the N3DV benchmark, it achieves a state-of-the-art PSNR of 32.56 dB.
- Artifact Reduction: Qualitative results show that ATGS preserves fine-grained details (such as human hands and fast-moving basketballs) without the severe blurring or missing body parts common in streaming or deformation-based baselines.
Limitations and Future Work
Despite its strong performance, the authors note a few limitations:
- Offline Paradigm: ATGS is an offline reconstruction method and does not currently support real-time training or streaming processing.
- COLMAP Dependency: The framework relies on COLMAP for initial camera poses and sparse point clouds, which introduces a heavy computational bottleneck during preprocessing.
- Sparse View Jitter: In regions with extremely sparse camera coverage (e.g., distant background audience areas), mild temporal jitter can still occur due to under-constrained optimization. Incorporating stronger temporal regularizers or geometric priors remains an open direction for future research.
Summary updated