FastFrames: Video Generation at Interactive Latency

Gene Chou, Guandao Yang, Fujun Luan, Lars Mescheder, Wei Dong, Stephan Richter, Vladlen KoltunarXiv 2026 · arXiv

TL;DR

FastFrames is a new family of autoregressive video diffusion models designed for "interactive latency"—responding to user inputs in under 50 milliseconds. By replacing heavy "accessory" components like VAE decoders and KV cache updates with lightweight alternatives, the researchers achieved up to a 10x speedup over state-of-the-art baselines without sacrificing visual quality.

What Problem Does This Paper Address?

While recent video diffusion models have become incredibly realistic, they remain too slow for truly interactive applications like gaming or real-time world simulation. The authors argue that the standard metric, Frames Per Second (FPS), is misleading for interactivity. A model might generate 30 FPS in a single batch, but if the user has to wait a full second for that batch to process, the experience feels laggy.

True interactivity requires low input-to-response latency. Human-computer interaction research suggests a threshold of roughly 50–60ms for a system to feel "instant." Existing autoregressive models, which generate video frame-by-frame, typically take hundreds of milliseconds per frame, falling well short of this "interactive regime."

Methods

The authors began by profiling the inference pipeline of a standard 1.3B parameter model. They discovered a surprising "aha moment": the actual diffusion denoising steps weren't the main bottleneck. Instead, "accessory" operations—specifically VAE decoding and KV cache updates—accounted for over half of the total latency.

To solve this, FastFrames introduces three main innovations:

  1. Lightweight Pixel Decoder: Traditional VAE decoders are computationally expensive because they repeatedly upsample features using dense 3D convolutions. FastFrames replaces this with a ConvNeXt-based "readout" head that operates at low resolution and uses a single linear layer to "unpatchify" latents into pixels. This reduces decoding latency by 147x (from 147ms to just 1ms).
  2. Faster KV Cache Updates: Autoregressive models need to update their Key-Value (KV) cache for every new frame to maintain temporal consistency. Baselines usually run a full, expensive model pass just for this update. FastFrames replaces this with a tiny "repair" network (only 4 transformer blocks) that fixes the noisy cache entries 10x faster than the baseline.
  3. System-Level Optimizations: The team implemented hardware-aware optimizations, including torch.compile, fused kernels, and Context Parallelism (CP) to efficiently distribute computation across multiple GPUs.
The FastFrames pipeline (bottom) replaces the slow VAE decoder and KV cache write steps of the baseline (top) with lightweight alternatives, achieving a 7.6x total speedup.

Datasets

  • Training: The pixel decoder was trained on an internal dataset of 10K real videos. The authors used a frozen DINOv3 backbone as a discriminator to ensure the new decoder maintained high visual sharpness and realism.
  • Evaluation: The models were benchmarked using VBench, a comprehensive suite for video generative models, to compare quality, semantics, and style against existing state-of-the-art models.
  • Interactive Tasks: The authors adapted their techniques to existing interactive models like LingBot (camera-conditioned) and RealWonder (physics-conditioned) to demonstrate real-world speedups.

Key Insights and Ideas

  • The "Accessory" Bottleneck: In the era of few-step distillation (where we only need 1-2 diffusion steps), the components we used to think were "cheap" (like VAEs) have become the new primary bottlenecks.
  • Readout vs. Reconstruction: You don't need a general-purpose VAE decoder to get good pixels. Because the Diffusion Transformer (DiT) already produces rich, spatially contextualized features, a simple, specialized "readout" head can extract high-quality images much faster.
  • Repairing is Better than Re-running: You cannot simply skip KV cache refinement without causing visual "drift" and artifacts. However, you don't need the full power of the main model to fix the cache; a lightweight adapter can "repair" the noisy entries effectively.
Ablation study showing that while removing KV refinement causes artifacts (third/fourth columns), the FastFrames repair network (second column) matches the quality of the expensive baseline (first column).

Key Results

FastFrames establishes a new Pareto frontier for video generation, significantly outperforming baselines in the speed-vs-quality tradeoff.

  • FastFrames-S (1.3B parameters): Runs at 36ms per frame (28 responses per second) on a single NVIDIA B200 GPU. It matches the quality of the Causal Forcing++ baseline while being 7.6x faster.
  • FastFrames-L (14B parameters): Runs at 45ms per frame (22 responses per second) on an 8x B200 node, providing significantly higher realism and better text rendering than smaller models.
  • Interactive Speedups: When applied to off-the-shelf interactive models, FastFrames achieved a 16x speedup on LingBot and a 22x speedup on RealWonder, bringing both into the interactive regime for the first time.
Quality-speed tradeoff showing FastFrames is the only model family to enter the "Interactive Regime" (<50ms latency) while maintaining high VBench scores.

Limitations and Future Work

  • Resolution: The current interactive speeds are measured at 480p. Increasing the resolution to 720p or 4K significantly increases latency, requiring further optimization to stay within the 50ms budget.
  • Hardware Requirements: The results rely on high-end H100 or B200 GPUs. Making these models run interactively on consumer-grade or mobile devices remains a challenge.
  • Long-term Consistency: While excellent for short bursts of interaction, maintaining perfect physical plausibility and consistency over very long video sequences is still an open area for improvement.

Summary updated

Abstract

Autoregressive video diffusion models generate frames based on new observations and inputs, making them a promising medium for interactive visual experiences. However, existing models are orders of magnitude too slow for fluid interaction. This paper introduces FastFrames, a family of fast autoregressive video diffusion models that achieves interactive latency without compromising quality. We profile the inference pipeline of state-of-the-art models and find that operations often treated as fixed implementation details, such as VAE decoding and KV cache updates, in fact account for over half of end-to-end inference latency. We challenge the assumption that these components are indispensable and replace them with lightweight alternatives. Our 1.3B variant runs at 36 ms per frame, or 28 responses per second (RPS), on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8x slower. Our 14B variant runs at 45 ms per frame, or 22 RPS, on an 8x B200 GPU node and yields significantly better visual quality and realism. We show that FastFrames enables real-time interaction in applications such as camera-conditioned and physics-conditioned video generation.
BibTeX
@misc{chou2026fastframes,
  author = {Gene Chou and Guandao Yang and Fujun Luan and Lars Mescheder and Wei Dong and Stephan Richter and Vladlen Koltun},
  title = {{FastFrames: Video Generation at Interactive Latency}},
  year = {2026},
  eprint = {2611.23226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23226},
}
Figure