TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction

DaDong Jiang, Zhihui Ke, Xiaobo Zhou, Zhi Hou, Xianghui Yang, Wenbo Hu, Tie Qiu, Chunchao Guo

TL;DR

TimeFormer is a plug-and-play Transformer module that enhances deformable 3D Gaussian Splatting by implicitly modeling global, cross-time motion patterns during training. By utilizing a clever two-stream optimization strategy, it transfers this rich temporal knowledge directly into the base model's weights, significantly boosting reconstruction quality for complex motions and reflections while introducing absolutely zero computational overhead during inference.

What Problem Does This Paper Address?

Reconstructing dynamic 3D scenes from videos is a core challenge in computer vision. While 3D Gaussian Splatting (3DGS) has revolutionized static scene rendering, extending it to dynamic scenes usually relies on a "deformation field" to map moving points back to a static, canonical space over time.

However, existing deformation methods typically look at each timestamp independently or only consider immediate neighboring frames (e.g., using local optical flow constraints). This localized view makes it incredibly difficult to accurately reconstruct scenes with violent or sudden movements, complex geometries, or highly reflective surfaces, as the model lacks a holistic understanding of the scene's trajectory across the entire timeline.

Methods

The authors introduce TimeFormer, an auxiliary framework added during the training phase to capture long-term temporal relationships. It consists of two main components:

  1. Cross-Temporal Transformer Encoder: Inspired by the concept of treating sample relationships across a batch dimension, TimeFormer treats randomly sampled timestamps from a video sequence as a "time batch." It takes the canonical positions of the 3D Gaussians combined with time encodings and passes them through self-attention layers. This allows the network to adaptively aggregate motion information across the entire time series.
  2. Two-Stream Optimization Strategy: To prevent the Transformer from slowing down real-time rendering, TimeFormer is only used during training. The framework sets up two parallel pipelines: a standard deformation branch and a TimeFormer-assisted branch. Crucially, both branches share the same underlying deformation network weights. By optimizing joint losses, the temporal insights gained by the TimeFormer branch are baked directly into the shared deformation field. During inference, the TimeFormer module is completely discarded.
The overall framework of TimeFormer, showing how the auxiliary training module uses temporal attention to guide the shared deformation field.

Datasets

The authors evaluated TimeFormer across three distinct types of dynamic datasets:

  • N3DV Dataset: A multi-view video benchmark containing complex real-world dynamic scenes (e.g., cooking, flame salmon).
  • HyperNeRF Dataset: A challenging monocular video dataset featuring topologically varying and moving objects (e.g., cutting a lemon).
  • NeRF-DS Dataset: Specifically targeted at evaluating dynamic objects with highly specular and reflective surfaces (e.g., metallic cups, glass bottles).

Key Insights and Ideas

  • Time as a Batch Dimension: Instead of designing complex, hand-crafted geometric or physical motion priors, TimeFormer leverages standard self-attention across a batch of timestamps. This allows the model to learn global motion patterns purely from RGB data.
  • Global Gradient Flow: Mathematically, adding TimeFormer introduces an extra gradient term during backpropagation. This allows the optimization of a Gaussian's state at a specific timestamp to be directly influenced by its past and future states, smoothing out the optimization landscape.
  • The Efficiency Side-Effect: By guiding the model toward a more physically consistent canonical space, TimeFormer naturally eliminates redundant Gaussians. The resulting scene representation is significantly more compact, which unexpectedly accelerates rendering speeds during inference.
TimeFormer guides the model toward a more compact canonical space representation, resulting in fewer points and higher FPS.

Key Results

TimeFormer consistently outperformed baseline methods across all datasets without impacting inference speeds:

  • Multi-View (N3DV): Improved the Peak Signal-to-Noise Ratio (PSNR) of the baseline 4DGS by 0.74 dB and STGS by 0.61 dB.
  • Monocular (HyperNeRF): Achieved a PSNR boost of 0.89 dB for 4DGS and 0.94 dB for DeformGS.
  • Efficiency Gains: On the HyperNeRF "cut lemon" scene, adapting TimeFormer to DeformGS cut the number of required Gaussians by more than half (from 169.6k down to 82.9k) while nearly doubling the rendering speed from 30.1 FPS to 58.9 FPS.

Limitations and Future Work

While TimeFormer excels at capturing robust global motion and handling challenging reflections, the authors note a minor trade-off: the regularization effect of the temporal attention mechanism can occasionally cause the model to produce overly smooth textures on objects that possess highly intricate, fine-grained structural details. Future work will likely explore ways to better balance global temporal consistency with high-frequency spatial details.

Summary updated

Abstract

Dynamic scene reconstruction is a long-term challenge in 3D vision. Recent methods extend 3D Gaussian Splatting to dynamic scenes via additional deformation fields and apply explicit constraints like motion flow to guide the deformation. However, they learn motion changes from individual timestamps independently, making it challenging to reconstruct complex scenes, particularly when dealing with violent movement, extreme-shaped geometries, or reflective surfaces. To address the above issue, we design a plug-and-play module called TimeFormer to enable existing deformable 3D Gaussians reconstruction methods with the ability to implicitly model motion patterns from a learning perspective. Specifically, TimeFormer includes a Cross-Temporal Transformer Encoder, which adaptively learns the temporal relationships of deformable 3D Gaussians. Furthermore, we propose a two-stream optimization strategy that transfers the motion knowledge learned from TimeFormer to the base stream during the training phase. This allows us to remove TimeFormer during inference, thereby preserving the original rendering speed. Extensive experiments in the multi-view and monocular dynamic scenes validate qualitative and quantitative improvement brought by TimeFormer. Project Page: https://patrickddj.github.io/TimeFormer/
BibTeX
@inproceedings{jiang2025timeformer,
  author = {DaDong Jiang and Zhihui Ke and Xiaobo Zhou and Zhi Hou and Xianghui Yang and Wenbo Hu and Tie Qiu and Chunchao Guo},
  title = {{TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=67vcX2yXLA},
}
Figure