TL;DR
TimeFormer is a plug-and-play Transformer module that enhances deformable 3D Gaussian Splatting by implicitly modeling global, cross-time motion patterns during training. By utilizing a clever two-stream optimization strategy, it transfers this rich temporal knowledge directly into the base model's weights, significantly boosting reconstruction quality for complex motions and reflections while introducing absolutely zero computational overhead during inference.
What Problem Does This Paper Address?
Reconstructing dynamic 3D scenes from videos is a core challenge in computer vision. While 3D Gaussian Splatting (3DGS) has revolutionized static scene rendering, extending it to dynamic scenes usually relies on a "deformation field" to map moving points back to a static, canonical space over time.
However, existing deformation methods typically look at each timestamp independently or only consider immediate neighboring frames (e.g., using local optical flow constraints). This localized view makes it incredibly difficult to accurately reconstruct scenes with violent or sudden movements, complex geometries, or highly reflective surfaces, as the model lacks a holistic understanding of the scene's trajectory across the entire timeline.
Methods
The authors introduce TimeFormer, an auxiliary framework added during the training phase to capture long-term temporal relationships. It consists of two main components:
- Cross-Temporal Transformer Encoder: Inspired by the concept of treating sample relationships across a batch dimension, TimeFormer treats randomly sampled timestamps from a video sequence as a "time batch." It takes the canonical positions of the 3D Gaussians combined with time encodings and passes them through self-attention layers. This allows the network to adaptively aggregate motion information across the entire time series.
- Two-Stream Optimization Strategy: To prevent the Transformer from slowing down real-time rendering, TimeFormer is only used during training. The framework sets up two parallel pipelines: a standard deformation branch and a TimeFormer-assisted branch. Crucially, both branches share the same underlying deformation network weights. By optimizing joint losses, the temporal insights gained by the TimeFormer branch are baked directly into the shared deformation field. During inference, the TimeFormer module is completely discarded.
Datasets
The authors evaluated TimeFormer across three distinct types of dynamic datasets:
- N3DV Dataset: A multi-view video benchmark containing complex real-world dynamic scenes (e.g., cooking, flame salmon).
- HyperNeRF Dataset: A challenging monocular video dataset featuring topologically varying and moving objects (e.g., cutting a lemon).
- NeRF-DS Dataset: Specifically targeted at evaluating dynamic objects with highly specular and reflective surfaces (e.g., metallic cups, glass bottles).
Key Insights and Ideas
- Time as a Batch Dimension: Instead of designing complex, hand-crafted geometric or physical motion priors, TimeFormer leverages standard self-attention across a batch of timestamps. This allows the model to learn global motion patterns purely from RGB data.
- Global Gradient Flow: Mathematically, adding TimeFormer introduces an extra gradient term during backpropagation. This allows the optimization of a Gaussian's state at a specific timestamp to be directly influenced by its past and future states, smoothing out the optimization landscape.
- The Efficiency Side-Effect: By guiding the model toward a more physically consistent canonical space, TimeFormer naturally eliminates redundant Gaussians. The resulting scene representation is significantly more compact, which unexpectedly accelerates rendering speeds during inference.
Key Results
TimeFormer consistently outperformed baseline methods across all datasets without impacting inference speeds:
- Multi-View (N3DV): Improved the Peak Signal-to-Noise Ratio (PSNR) of the baseline 4DGS by 0.74 dB and STGS by 0.61 dB.
- Monocular (HyperNeRF): Achieved a PSNR boost of 0.89 dB for 4DGS and 0.94 dB for DeformGS.
- Efficiency Gains: On the HyperNeRF "cut lemon" scene, adapting TimeFormer to DeformGS cut the number of required Gaussians by more than half (from 169.6k down to 82.9k) while nearly doubling the rendering speed from 30.1 FPS to 58.9 FPS.
Limitations and Future Work
While TimeFormer excels at capturing robust global motion and handling challenging reflections, the authors note a minor trade-off: the regularization effect of the temporal attention mechanism can occasionally cause the model to produce overly smooth textures on objects that possess highly intricate, fine-grained structural details. Future work will likely explore ways to better balance global temporal consistency with high-frequency spatial details.
Summary updated