S2DiT: Sandwich Diffusion Transformer for Mobile Streaming Video Generation

Lin Zhao, Yushu Wu, Aleksei Lebedev, Dishani Lahiri, Meng Dong, Arpit Sahni, Michael Vasilkovsky, Hao Chen, Ju Hu, Aliaksandr Siarohin, Sergey Tulyakov, Yanzhi Wang, Anil Kag, Yanyu Li

TL;DR

This paper introduces S²DiT, a novel diffusion transformer architecture optimized for efficient, high-fidelity video generation directly on mobile devices. By combining specialized attention mechanisms and a unique "sandwich" design, S²DiT achieves state-of-the-art video quality while streaming at over 10 FPS on an iPhone, overcoming the computational limitations of previous models.

What Problem Does This Paper Address?

High-quality video generation models, particularly Diffusion Transformers (DiTs), have seen remarkable progress. However, their substantial computational demands and memory footprints make them impractical for real-time or on-device applications, especially on resource-constrained mobile hardware. This paper tackles the challenge of bridging this gap: how to achieve both high-fidelity and efficient streaming video generation simultaneously on mobile devices.

Methods

S²DiT introduces several key innovations:

  • Efficient Attention Mechanisms: To combat the quadratic complexity of standard self-attention, S²DiT employs two novel attention modules:
  • LinConv Hybrid Attention (LCHA): This module combines a learnable linear attention path with a depthwise 3D convolution path. This hybrid approach achieves linear complexity while preserving spatio-temporal fidelity, capturing both local details and global dependencies.
  • Stride Self-Attention (SSA): This module compresses intermediate feature maps by using strided operations, improving throughput and reducing computational cost.
  • Sandwich Architecture: The core architecture is a "sandwich" design that interleaves LCHA and SSA modules. The optimal placement and count of these modules are determined through a budget-aware dynamic programming search, optimizing for latency and memory constraints.
  • 2-in-1 Distillation Framework: To transfer the knowledge from massive teacher models (like Wan 2.2-14B) to the compact S²DiT student model, a two-stage distillation process is used:
  • Offline Cached Knowledge Distillation: This decouples teacher inference from student training by precomputing and caching teacher outputs (diffusion tuples, text embeddings). This significantly speeds up training and reduces peak memory usage.
  • Self-Forcing Distillation: This stage adapts the distilled model for streaming video generation using techniques like Distribution Matching Distillation (DMD) and self-forcing, enabling few-step auto-regressive generation.

Datasets

The paper uses a combination of public and internal datasets for training, totaling 300 million images and 50 million videos after quality filtering. For evaluation, they utilize the VBench score and metrics like CLIP score, FID, and FVD. They also mention using the OpenVid dataset for ablation studies.

Key Insights and Ideas

  • Hybrid Attention is Key: Simply using linear attention or only low-resolution attention (SSA) leads to a significant drop in quality. The combination of LCHA (for high-resolution detail) and SSA (for global context) is crucial for achieving both efficiency and fidelity.
  • Budget-Aware Search is Essential: The "sandwich" architecture isn't just a fixed pattern; it's dynamically discovered through a budget-aware search. This allows for optimal allocation of LCHA and SSA blocks based on specific mobile hardware constraints.
  • Offline Distillation is a Game-Changer for Mobile: The computational cost of distilling large models is a major bottleneck. The proposed offline caching strategy makes it feasible to transfer the quality of billion-parameter models to compact mobile models without prohibitive training costs.
  • Streaming Generation is Achievable on Mobile: The paper demonstrates that high-quality, real-time streaming video generation is no longer confined to powerful servers, but can be achieved on consumer mobile devices.

Key Results

  • Quality: S²DiT achieves video quality comparable to state-of-the-art server-side video generation models like Wan 2.1-1.3B and Hunyuan-13B, as evidenced by VBench scores (e.g., S²DiT-KD at 83.62).
  • Speed: The S²DiT-AR (auto-regressive) variant streams at over 10 FPS on an iPhone 16 Pro Max, a significant achievement for on-device video generation.
  • Efficiency: The model is designed to be mobile-efficient, with a compact decoder that achieves low latency and minimal quality degradation.
  • Ablation Studies: The paper shows that each component of LCHA is effective, and the sandwich architecture outperforms simpler hourglass designs.

Limitations and Future Work

  • Mobile Deployment Challenges: While S²DiT significantly reduces computational load, the KV-cache for SSA can still grow with the number of generated frames, leading to memory overhead. The paper addresses this with window attention for the KV-cache.
  • Further Optimization: The authors suggest that further exploration of efficient attention mechanisms and distillation strategies could lead to even greater improvements.
  • Broader Applications: While focused on mobile streaming, the efficient architecture and distillation techniques could be applied to other resource-constrained environments.

Summary updated

Abstract

Diffusion Transformers (DiTs) have recently improved video generation quality. However, their heavy computational cost makes real-time or on-device generation infeasible. In this work, we introduce S2DiT, a Streaming Sandwich Diffusion Transformer designed for efficient, high-fidelity, and streaming video generation on mobile hardware. S2DiT generates more tokens but maintains efficiency with novel efficient attentions: a mixture of LinConv Hybrid Attention (LCHA) and Stride Self-Attention (SSA). Based on this, we uncover the sandwich design via a budget-aware dynamic programming search, achieving superior quality and efficiency. We further propose a 2-in-1 distillation framework that transfers the capacity of large teacher models (e.g., Wan 2.2-14B) to the compact few-step sandwich model. Together, S2DiT achieves quality on par with state-of-the-art server video models, while streaming at over 10 FPS on an iPhone.
BibTeX
@inproceedings{zhao2026s,
  author = {Lin Zhao and Yushu Wu and Aleksei Lebedev and Dishani Lahiri and Meng Dong and Arpit Sahni and Michael Vasilkovsky and Hao Chen and Ju Hu and Aliaksandr Siarohin and Sergey Tulyakov and Yanzhi Wang and Anil Kag and Yanyu Li},
  title = {{S2DiT: Sandwich Diffusion Transformer for Mobile Streaming Video Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=rL0ynrQ6j0},
}
Figure