TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction

Dadong Jiang, Zhihui Ke, Xiaobo Zhou, Zhi Hou, Xianghui Yang, Wenbo Hu, Tie Qiu, Chunchao Guo

TL;DR

This paper introduces TimeFormer, a novel Transformer-based module that enhances dynamic 3D scene reconstruction by explicitly modeling temporal relationships between frames. By learning these relationships implicitly, TimeFormer allows existing 3D Gaussian Splatting methods to achieve more robust reconstructions of complex dynamic scenes, all while maintaining real-time inference speeds through a clever two-stream training strategy.

What Problem Does This Paper Address?

Reconstructing dynamic 3D scenes—think of a person dancing, a car driving, or objects interacting—is a notoriously difficult problem in computer vision. While recent advancements like 3D Gaussian Splatting (3DGS) have revolutionized static scene rendering, extending them to dynamic scenes presents a significant challenge. Existing methods often treat each moment in time independently, leading to difficulties in capturing complex motions, sudden changes, or scenes with reflective surfaces. They struggle to understand how a scene evolves over time, often relying on explicit motion cues or local temporal constraints that fail to capture long-range temporal dependencies. This leads to artifacts and inaccuracies, especially in challenging scenarios.

Methods

The core innovation is TimeFormer, a plug-and-play module designed to be integrated into existing deformable 3D Gaussian Splatting frameworks. TimeFormer's key components are:

  • Cross-Temporal Transformer Encoder: This module takes a sequence of sampled 3D Gaussians across different timestamps and uses a self-attention mechanism to implicitly learn their temporal relationships. It essentially looks at how Gaussians evolve over time, capturing subtle motion patterns that individual frame processing misses.
  • Two-Stream Optimization Strategy: To avoid the computational overhead of running TimeFormer during inference (which would slow down real-time rendering), the authors propose a training strategy. They train two parallel branches: the original deformation branch and a new branch incorporating TimeFormer. Crucially, they share the weights of the deformation fields between these two branches. This allows the motion knowledge learned by TimeFormer to be transferred to the original deformation branch. During inference, TimeFormer can be discarded, and the enhanced deformation field can be used, preserving the original rendering speed.

Datasets

The authors evaluated TimeFormer on several established dynamic scene datasets:

  • Multi-view datasets: N3DV Dataset [25]
  • Monocular datasets: HyperNeRF Dataset [39]
  • Dynamic scenes with specular surfaces: NeRF-DS Dataset [65]

These datasets cover a range of challenges, including complex motion, varying viewpoints, and reflective materials.

Key Insights and Ideas

  • Implicit Temporal Modeling is Key: The paper argues that explicitly modeling temporal relationships implicitly through a Transformer is more effective than relying on local temporal constraints or explicit motion cues. This allows the model to learn more nuanced and robust motion patterns.
  • Transformer for Temporal Dynamics: Applying Transformer architectures, which excel at capturing long-range dependencies in sequences, to the temporal dimension of 3D Gaussians is a powerful idea for dynamic scene reconstruction.
  • Training-Inference Decoupling: The two-stream optimization with shared weights is a critical insight. It allows the benefits of a complex temporal modeling module (TimeFormer) to be realized during training without sacrificing real-time performance during inference. This is a common challenge in many advanced computer vision techniques.
  • Faster Convergence and Better Canonical Space: TimeFormer not only improves reconstruction quality but also appears to guide the optimization process towards a more efficient canonical representation of the scene, leading to faster convergence and potentially fewer Gaussians needed for representation.

Key Results

  • Improved Reconstruction Quality: TimeFormer consistently improved the PSNR and SSIM metrics across various baseline methods (4DGS, STGS, DeformGS) on both multi-view and monocular datasets. For example, it achieved improvements of up to 0.89 PSNR and 0.019 SSIM on the HyperNeRF dataset.
  • Robustness to Complex Scenarios: Qualitative results show TimeFormer effectively handles challenging scenarios like violent movements (broom example), specular surfaces (metallic cup, spray gun), and distinguishing between moving and static objects.
  • Maintained Real-time Performance: The two-stream training strategy ensures that TimeFormer does not add any computational overhead during inference, preserving the high FPS characteristic of 3DGS methods. In some cases, it even led to higher FPS due to a more efficient canonical space.
  • Ablation Studies: The paper demonstrates that TimeFormer's performance is relatively insensitive to the number of transformer layers or batch size, suggesting a robust design. However, removing the shared deformation fields significantly degraded performance.

Limitations and Future Work

  • Overly Smooth Textures: The authors acknowledge that TimeFormer may produce overly smooth textures for objects with intricate details. This suggests a potential trade-off between temporal consistency and fine-grained texture detail.
  • Future Work: The paper implicitly suggests exploring ways to further refine texture reconstruction and potentially extend TimeFormer to even more complex dynamic scene phenomena. The success on specular surfaces also opens doors for more robust handling of challenging material properties in dynamic scenes.

Summary updated

Abstract

Dynamic scene reconstruction is a long-term challenge in 3D vision. Recent methods extend 3D Gaussian Splatting to dynamic scenes via additional deformation fields and apply explicit constraints like motion flow to guide the deformation. However, they learn motion changes from individual timestamps independently, making it challenging to reconstruct complex scenes, particularly when dealing with violent movement, extreme-shaped geometries, or reflective surfaces. To address the above issue, we design a plug-and-play module called TimeFormer to enable existing deformable 3D Gaussians reconstruction methods with the ability to implicitly model motion patterns from a learning perspective. Specifically, TimeFormer includes a Cross-Temporal Transformer Encoder, which adaptively learns the temporal relationships of deformable 3D Gaussians. Furthermore, we propose a two-stream optimization strategy that transfers the motion knowledge learned from TimeFormer to the base stream during the training phase. This allows us to remove TimeFormer during inference, thereby preserving the original rendering speed. Extensive experiments in the multi-view and monocular dynamic scenes validate qualitative and quantitative improvement brought by TimeFormer. Project Page: https://patrickddj.github.io/TimeFormer/
BibTeX
@inproceedings{jiang2025timeformer,
  author = {Dadong Jiang and Zhihui Ke and Xiaobo Zhou and Zhi Hou and Xianghui Yang and Wenbo Hu and Tie Qiu and Chunchao Guo},
  title = {{TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=67vcX2yXLA},
}
Figure