LSTD: Long Short-Term Temporal Diffusion for Video Generation

Haoyu Zhao, Jiaxi Gu, Shicong Wang, Tianyi Lu, Xing Zhang, Zuxuan Wu, Hang Xu, Yu-Gang JiangIEEE Trans. Multimedia 2026 · DOI

Abstract

Recently, text-driven video generation has achieved tremendous progress. However, existing methods neglect the contexts of long short-term frames in the video, thereby compromising temporal consistency. They also encounter challenges of heavy memory costs due to the use of the standard temporal attention mechanism and misalignment between training videos and captions. Additionally, previous approaches for long video generation are flawed because they are hard to ensure content diversity and consistency. To alleviate these issues, we propose a novel Long Short-term Temporal Diffusion (LSTD) model to generate videos with superior temporal consistency. We introduce two novel temporal modules, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i>, the Short-term Temporal Convolution and the Long-term Temporal Attention. The former can learn short-term features with a shallow structure, and the latter concentrates on long-term information of complex motion with a new memory-efficient attention mechanism. The combination of the two modules can ensure the temporal consistency of the generated videos. Furthermore, a novel inference method for long video generation is also proposed, which can iteratively generate hundreds of video frames. Experimental results on UCF-101, MSR-VTT, and two long video benchmarks prove that our method achieves superior zero-shot inference performance even when the size of the training data is reduced by 26.5 times.
BibTeX
@article{zhao2026lstd,
  author = {Haoyu Zhao and Jiaxi Gu and Shicong Wang and Tianyi Lu and Xing Zhang and Zuxuan Wu and Hang Xu and Yu-Gang Jiang},
  title = {{LSTD: Long Short-Term Temporal Diffusion for Video Generation}},
  journal = {IEEE Transactions on Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2026.3651052},
}

Loading related content...

Figure