MPEG-Diffusion: Mixed-Resolution Video Generation

Dennis Menn, Tianpei Gu, Guande He, Aryan V S, Yanke Song, Nan Liu, Desai Xie, Xun Huang, Chenfeng Xu, Diana MarculescuarXiv 2026 · arXiv

TL;DR

MPEG-Diffusion is a novel framework that slashes the computational cost of video diffusion transformers (DiTs) by focusing high-resolution generation only on visually important regions. By introducing dual-resolution attention and a specialized latent-space decoder, it achieves up to a 4.8× speedup while maintaining quality nearly indistinguishable from full-resolution generation.

What Problem Does This Paper Address?

State-of-the-art video generation models, like Video Diffusion Transformers (DiTs), produce stunning results but are notoriously compute-hungry. The primary bottleneck is the attention mechanism, which scales quadratically with the number of tokens. Currently, these models spend compute uniformly: a clear blue sky receives the same high-resolution processing as a detailed human face.

While "mixed-resolution" generation—generating important areas at high resolution (HR) and backgrounds at low resolution (LR)—seems like an obvious solution, it has historically failed for two reasons:

  1. Quality Degradation: Previous methods suffered from blurry details and jarring "seams" where the HR and LR regions met.
  2. Prior Knowledge Requirement: Most methods require knowing the HR region (e.g., where the subject is) in advance, which is impossible for text-to-video tasks where the video doesn't exist yet.

Methods

MPEG-Diffusion addresses these issues through three core components, illustrated in the system overview:

Overview of the MPEG-Diffusion architecture, highlighting dual-resolution attention, the self-selected HR region, and the mixed-res decoder.
  1. Dual-Resolution Attention: The authors identify the "curse of mixed resolution," proving that a single LR token cannot mathematically match the attention impact of multiple HR tokens if they share the same Keys and Values (K/V). To fix this, they use linear maps to translate K/V between resolutions. This allows each query to "read" the entire video at its own native resolution, avoiding cross-resolution distortion.
  2. Mixed-Resolution Decoder: Standard VAE decoders are sensitive to neighboring patches. If you decode HR and LR latents separately and paste them together, you get visible seams. MPEG-Diffusion uses a small 3D convolutional "mixer" to blend the HR and LR latents in latent space before the final decoding step, ensuring spatial consistency.
  3. Self-Selected HR Region: Instead of relying on an external detector, the authors found that DiT features naturally "know" where the subject is early in the denoising process. They add a simple linear head to the DiT to predict the HR mask on the fly, enabling fully automated text-to-video generation.

Datasets

The researchers evaluated their method across several high-end models and benchmarks:

  • Models: Wan2.1 (1.3B and 14B parameters) for video, and FLUX.2 [klein] 4B for image generation.
  • Benchmarks: 1,003 prompts from MovieGen and 944 prompts from VBench.
  • Training Data: The specialized components (mixer and region head) were trained on approximately 30,000 real video clips and 2,500 labeled clips.

Key Insights and Ideas

  • The Attention Mismatch: A key "aha moment" was the mathematical proof that sharing K/V across resolutions is fundamentally flawed. The strong linear relationship found between HR and LR tokens in pre-trained models (with cosine similarities up to 0.96) made the dual-resolution mapping approach highly effective.
  • DiTs as Implicit Trackers: The discovery that a single linear layer on frozen DiT features can accurately track a subject across 80 frames is a testament to the rich spatial-temporal information already present in these models.
  • Latent Blending vs. Pixel Pasting: The paper demonstrates that the "seam" problem isn't just about resolution; it's about the VAE decoder's dependency on local context. Blending in the latent space is essential for visual fidelity.
Comparison showing how the mixer removes the seam across the HR/LR boundary compared to naive resize-and-paste methods.

Key Results

MPEG-Diffusion sets a new standard for efficient video generation:

  • Speed: It achieves a 4.8× speedup on the Wan2.1 14B model by processing only 30% of the tokens at full resolution.
  • Quality: In blind user studies, raters preferred MPEG-Diffusion over existing mixed-resolution baselines (like Foveated Diffusion) in over 90% of cases.
  • Near-Lossless: Against full-resolution generation, MPEG-Diffusion achieved a 45.2% win rate, meaning human raters found it nearly indistinguishable from the much slower full-res version.
Figure 1. MPEG-Diffusion generates the important region at high resolution and the rest at low resolution: on average 4.8× faster (30% token use) at near full-resolution quality, with a DiT self-selected high-res region. Prompt: a close-up of a rose with morning droplets.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Structural Constraints: The method assumes a linear relationship between HR and LR tokens. While strong, this is still a constraint that might lead to minor quality trade-offs in complex scenes.
  • Visual Contrast: In some cases, the transition between a very sharp subject and a blurry background might feel stylistically inconsistent.
  • End-to-End Training: Currently, the region selection head is trained on labels from external models (LLMs and SAM). Integrating this into an end-to-end training pipeline could further optimize performance.

Summary updated

Abstract

Video diffusion transformers (DiTs) generate realistic videos, but their heavy computational cost limits broader real-world applications. Mixed-resolution generation reduces this cost by generating the visually important region at high resolution (HR) and the rest at low resolution (LR), yet existing methods suffer a severe quality drop on videos and require knowing the HR region in advance. We show that these failures are not inherent to mixed resolution, and propose $\textbf{MPEG-Diffusion}$ to address their three root causes: (1) dual-resolution attention to mitigate the curse of mixed resolution, where each query reads the K/V of its own resolution, (2) a mixed-resolution decoder to blend the HR and LR latents before decoding and resolve distortion, and (3) a self-selected HR region using the DiT's knowledge to find the visually important region without requiring it in advance. Processing only $30\%$ of full-resolution tokens, MPEG-Diffusion speeds up denoising by $4.8\times$ on Wan2.1 14B ($3.8\times$ on 1.3B), outperforming prior methods on nearly every metric. In the user study, we demonstrate the high quality of MPEG-Diffusion, with raters preferring it over prior mixed-resolution methods in over $90\%$ of pairs and rating it close to full resolution.
BibTeX
@misc{menn2026mpeg,
  author = {Dennis Menn and Tianpei Gu and Guande He and Aryan V S and Yanke Song and Nan Liu and Desai Xie and Xun Huang and Chenfeng Xu and Diana Marculescu},
  title = {{MPEG-Diffusion: Mixed-Resolution Video Generation}},
  year = {2026},
  eprint = {2611.12069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12069},
}
Figure