Mixture of Contexts for Long Video Generation

Shengqu Cai, Ceyuan Yang, Lvmin Zhang, Yuwei Guo, Junfei Xiao, Ziyan Yang, Yinghao Xu, Zhenheng Yang, Alan Yuille, Leonidas Guibas, Maneesh Agrawala, Lu Jiang, Gordon Wetzstein

TL;DR

Generating minutes-long videos with Diffusion Transformers (DiTs) is notoriously difficult due to the quadratic memory and computational wall of self-attention. This paper introduces Mixture of Contexts (MoC), a method that recasts long-video generation as an internal information retrieval task where tokens dynamically select only the most relevant historical chunks to attend to. By pruning over 85% of the context, MoC slashes attention FLOPs by over 7× and delivers a 2.2× end-to-end speedup while actually improving motion dynamics and maintaining long-term consistency.

What Problem Does This Paper Address?

State-of-the-art video generation models rely heavily on Transformer-based diffusion architectures. However, scaling these models to generate minute- or hour-long videos exposes a fundamental bottleneck: standard self-attention scales quadratically, $O(L^2)$, with sequence length. For a 1-minute, 480p video, the sequence length easily reaches 180,000 tokens, making dense attention computationally intractable.

Prior attempts to solve this either compress past frames into lossy, compact representations (which discard fine details) or enforce fixed, static sparse attention patterns (which cannot adapt to dynamic scene changes or shot cuts). The core challenge isn't just saving compute; it's learning how to selectively recall the right context at the right time to preserve character identities, actions, and backgrounds across extended timelines.

Methods

The authors propose Mixture of Contexts (MoC), a learnable sparse attention routing module that acts as a long-term memory retrieval engine.

Overview of the Adaptive Mixture of Contexts routing pipeline.

Key components of the architecture include:

  1. Content-aligned Chunking: Instead of slicing the token stream into arbitrary uniform windows, MoC partitions the multi-modal stream along natural boundaries—individual frames, distinct shots, and text captions.
  2. Dynamic Top-$k$ Routing: For each chunk, the keys are mean-pooled into a single representative descriptor vector. Every query token computes a dot-product similarity against these descriptors, dynamically selecting only the top-$k$ most relevant chunks to attend to.
  3. Mandatory Anchors: To stabilize local fidelity and prevent prompt drift, every query token is forced to attend to two anchors: all text tokens (acting as an "attention sink") and its own intra-shot local window.
  4. Causal Routing Mask: The authors apply a causal mask during the routing stage. This restricts chunks to only attend to past context, transforming the routing graph into a Directed Acyclic Graph (DAG) and preventing pathological feedback loops.
  5. Routing Regularization: To prevent a "dead expert" problem (where the router repeatedly picks the same chunks), they introduce Context Drop-off (randomly masking out a subset of chosen chunks) and Context Drop-in (randomly injecting extraneous chunks) during training.

Datasets

  • Training: The model is trained on a large-scale, scene-level multi-shot dataset curated from public narrative videos (movies, TV shows, and documentaries) following the data preparation protocol of Long-Context Tuning (LCT). It contains approximately 500K annotated scenes (averaging 5 shots per scene, totaling ~2.5M clips), augmented with 1M long single-shot videos segmented into sub-shots.
  • Evaluation: Quantitative evaluation is conducted using the popular VBench benchmark, focusing on metrics like subject consistency, background consistency, motion smoothness, and dynamic degree.

Key Insights and Ideas

  • The Power of Mean-Pooling: Simple arithmetic mean-pooling of video chunks works remarkably well as a routing descriptor. Because adjacent spatiotemporal video tokens are highly redundant, their first principal component typically explains over 90% of local variance. A simple average effectively captures dominant semantics while filtering out high-frequency noise.
  • Causality Prevents Loop Closures: Without a causal mask on the sparse router, models easily fall into pathological feedback loops (e.g., Shot 9 strongly routes to Shot 11, while Shot 11 simultaneously routes back to Shot 9). This traps the model in a two-node cycle, causing frozen motion or identity drift. Enforcing a DAG structure eliminates this completely.
Illustration of how omitting causal masking creates pathological feedback loops, trapping the model and causing visual stagnation.
  • Sparsity Enhances Motion: Interestingly, sparsification doesn't just save compute—it reallocates the model's attention budget away from redundant frames and toward salient visual events. This shift significantly boosts the generated video's motion diversity and dynamic range.

Key Results

  • Computational Efficiency: MoC prunes over 85% of token pairs, reducing attention FLOPs by more than 7×. This translates to a measured 2.2× end-to-end generation speedup on minute-scale scenes (~180k tokens).
Performance benchmark comparing MoC to full attention, demonstrating near-linear scaling in FLOPs and latency as the number of shots scales.
  • Improved Video Quality: On the VBench benchmark, MoC outperforms the dense baseline (LCT) in Motion Smoothness (0.9920 vs. 0.9859) and substantially improves Dynamic Degree (0.5625 vs. 0.4583), while maintaining high subject and background consistency.
  • Zero-Shot Generalization: When plugged directly into a pretrained dense DiT model without any fine-tuning, MoC still preserves subject identity and background layout reasonably well, proving that the mean-pooled retrieval signal is inherently robust.

Limitations and Future Work

While MoC achieves massive FLOP reductions, the current runtime implementation still relies on general-purpose variable-length attention kernels and framework-level gathers. The authors note that substantial headroom remains for further speedups through hardware-software co-design. Developing customized CUDA or Triton kernels optimized specifically for chunk-aware, block-sparse attention and fused routing-attention operators is highlighted as a primary direction for future research.

Summary updated

Abstract

Long video generation is fundamentally a long context memory problem: models must retain and retrieve salient events across a long range without collapsing or drifting. However, scaling diffusion transformers to generate long-context videos is fundamentally limited by the quadratic cost of self-attention, which makes memory and computation intractable and difficult to optimize for long sequences. We recast long-context video generation as an internal information retrieval task and propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine. In MoC, each query dynamically selects a few informative chunks plus mandatory anchors (caption, local windows) to attend to, with causal routing that prevents loop closures. As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content. Efficiency follows as a byproduct of retrieval (near-linear scaling), which enables practical training and synthesis, and the emergence of memory and consistency at the scale of minutes.
BibTeX
@inproceedings{cai2026mixture,
  author = {Shengqu Cai and Ceyuan Yang and Lvmin Zhang and Yuwei Guo and Junfei Xiao and Ziyan Yang and Yinghao Xu and Zhenheng Yang and Alan Yuille and Leonidas Guibas and Maneesh Agrawala and Lu Jiang and Gordon Wetzstein},
  title = {{Mixture of Contexts for Long Video Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=y6XJZlEC2x},
}
Figure