SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation

Kehong Gong, Zhengyu Wen, Mingxi Xu, Weixia He, Qi Wang, Ning Zhang, Zhengyu Li, Chenbin Li, Dongze Lian, Wei Zhao, Xiaoyu He, Mingyuan ZhangarXiv 2025 · DOI · arXiv

Introduction

Reconstructing dynamic 3D objects—known as 4D generation—from monocular video remains a significant challenge in computer vision. Current approaches face three primary hurdles: a severe scarcity of high-quality 4D mesh datasets, a difficult trade-off between generalization and temporal consistency, and high architectural complexity that often leads to "forgetting" the robust 3D priors learned from static datasets.

SWiT-4D addresses these issues by introducing a minimalistic, parameter-free extension for 3D Diffusion Transformers (DiTs). This approach transforms a static 3D generator into a dynamic 4D one without adding a single new parameter to the transformer backbone, preserving the generalization power of pre-trained models while achieving remarkable temporal stability.

Methodology

The core of the SWiT-4D framework is the Sliding-Window Transformer mechanism. Rather than processing video frames in isolation, the model analyzes a local window of frames to ensure temporal coherence.

Architecture comparison of conventional single-frame solutions versus the proposed SWiT-4D, highlighting the sliding-window mechanism and 1D RoPE.

Key technical components include:

  • 1D Rotary Positional Encoding (RoPE): Applied along the temporal axis, RoPE allows the model to understand relative timing between frames. The authors demonstrate a "Lossless Temporal Property": when the window size is one, the operations collapse back into the original single-frame 3D model, ensuring pre-trained knowledge is perfectly preserved.
  • Temporal Residual Learning: The attention mechanism is decomposed into a static intra-frame prior and a temporal inter-frame residual. This encourages the model to focus on learning motion dynamics rather than re-learning object geometry.
  • Trajectory Optimization: A lightweight module uses a differentiable renderer to align the generated mesh's mask with the video's foreground, recovering the object's global world-space translation.

Datasets and Evaluation

The model was trained and evaluated using several diverse datasets:

  • Truebones Zoo: A curated collection of over 1,000 animal motion sequences used for primary training.
  • Objaverse: Utilized for zero-shot evaluation to test generalization across unseen object categories.
  • Consistent4D & In-the-wild videos: Used to verify robustness against unconstrained, real-world footage.

Key Results and Insights

SWiT-4D establishes a new state-of-the-art for video-to-4D generation. On the Truebones Zoo dataset, it reduced the Chamfer Distance (3D error) by over 2.5x compared to the TripoSG baseline. It also significantly outperformed existing methods like L4GM and GVFD in temporal smoothness metrics.

Qualitative comparison of SWiT-4D against baseline methods on dynamic sequences, demonstrating superior geometry and temporal consistency.

One of the most striking findings is the model's data efficiency. By fine-tuning on as little as a single 10-second video clip, the model learns enough temporal logic to improve performance across various species. Furthermore, the use of RoPE makes the model shift-equivariant, allowing it to handle videos of arbitrary length during inference without absolute timestamp constraints.

Limitations and Future Work

While SWiT-4D represents a major advancement, it currently assumes a static camera. Future iterations could integrate SLAM or camera-tracking techniques to handle dynamic camera movements. Additionally, while the model excels at single-object reconstruction, multi-object scenes with complex occlusions remain a frontier for future research. Enhancing the temporal consistency of textures and materials across long sequences is another area for potential improvement.

Summary updated

Abstract

Despite significant progress in 4D content generation, the conversion of monocular videos into high-quality animated 3D assets with explicit 4D meshes remains considerably challenging. The scarcity of large-scale, naturally captured 4D mesh datasets further limits the ability to train generalizable video-to-4D models from scratch in a purely data-driven manner. Meanwhile, advances in image-to-3D generation, supported by extensive datasets, offer powerful prior models that can be leveraged. To better utilize these priors while minimizing reliance on 4D supervision, we introduce SWiT-4D, a Sliding-Window Transformer for lossless, parameter-free temporal 4D mesh generation. SWiT-4D integrates seamlessly with any Diffusion Transformer (DiT)-based image-to-3D generator, adding spatial-temporal modeling across video frames while preserving the original single-image forward process, enabling 4D mesh reconstruction from videos of arbitrary length. To recover global translation, we further introduce an optimization-based trajectory module tailored for static-camera monocular videos. SWiT-4D demonstrates strong data efficiency: with only a single short (<10s) video for fine-tuning, it achieves high-fidelity geometry and stable temporal consistency, indicating practical deployability under extremely limited 4D supervision. Comprehensive experiments on both in-domain zoo-test sets and challenging out-of-domain benchmarks (C4D, Objaverse, and in-the-wild videos) show that SWiT-4D consistently outperforms existing baselines in temporal smoothness. Project page: https://animotionlab.github.io/SWIT4D/
BibTeX
@misc{gong2025swit,
  author = {Kehong Gong and Zhengyu Wen and Mingxi Xu and Weixia He and Qi Wang and Ning Zhang and Zhengyu Li and Chenbin Li and Dongze Lian and Wei Zhao and Xiaoyu He and Mingyuan Zhang},
  title = {{SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation}},
  year = {2025},
  eprint = {2512.10860},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.10860},
}
Figure