Scaling High Resolution Video Generation Beyond Memory Bottlenecks

441 concepts372 papers1 proposal

Generating a static image is a two-dimensional problem. Generating a video introduces a punishing third dimension: time ($T$). When scaling to high-definition resolutions like 1080p or 4K, the data volume explodes exponentially. A single high-resolution video clip contains millions of pixels across dozens of frames. Processing this volume of data holistically causes standard generative architectures—particularly those relying on self-attention—to hit a hard hardware wall. Because the memory required by standard attention mechanisms scales quadratically with the number of tokens ($H \times W \times T$), high-resolution video synthesis quickly exceeds the VRAM limits of modern graphics hardware.

To make high-resolution video synthesis practical on consumer-grade or standard research hardware, researchers have shifted away from brute-force computation. Instead, they employ architectural strategies that exploit the spatial and temporal redundancies of video data. These approaches can be distilled into three core paradigms: coarse-to-fine cascades, localized windowing, and sub-quadratic attention guidance.

Coarse-to-Fine Cascades and Latent Space Factorization

Why process raw, high-resolution pixels from the very beginning of the generation process when you can operate in a compressed or simplified space? The first core intuition for overcoming the memory bottleneck is to decouple semantic layout generation from high-resolution detail synthesis.

Castrejón et al. pioneered this structural decomposition with Cascaded Video Generation for Videos In-the-Wild. Instead of generating a high-resolution sequence in a single pass, the framework breaks the task into a hierarchy. The model first generates a low-resolution "sketch" that establishes the global motion and structure. It then progressively upscales this sketch through greedy, stage-by-stage training. This prevents the model from having to hold the entire high-resolution sequence's computational graph in memory simultaneously.

A similar division of labor is utilized by Zhang et al. in Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation. Pixel-based video models excel at understanding text prompts and ensuring initial motion alignment, but they are too computationally expensive to run at high resolutions. The authors resolve this by using a pixel-based model exclusively for low-resolution generation, then handing the output over to an efficient latent-based diffusion model for high-resolution upscaling.

By shifting the heavy computational lifting into a projected latent space—as explored by Yu et al. in Video Probabilistic Diffusion Models in Projected Latent Space—the model manipulates compact mathematical representations rather than raw pixel grids, drastically lowering the baseline memory footprint.

Divide and Denoise via Localized Windows

Even within an optimized latent space, ultra-high-resolution canvases (such as 4K frames or 360° panoramas) will overwhelm GPU memory if the model attempts to compute global dependencies across the entire frame at once. The solution is a "divide-and-conquer" approach: breaking the canvas into smaller spatial or temporal patches, processing them independently, and using specialized blending mechanics to ensure global consistency.

In DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes, Liu et al. introduced a "rotating window" denoising strategy to generate infinite-length, high-resolution panoramic videos. Instead of calculating attention across the entire 360° canvas simultaneously, the model restricts its attention mechanism to a localized window. Crucially, this window shifts and rotates across successive denoising steps. Because the windows overlap, contextual information propagates across the boundaries over time, resulting in a seamless panoramic output while keeping VRAM consumption completely constant.

Brute-Force Attention (High VRAM):
[================== Full Canvas ==================] -> Computes all-to-all dependencies

Rotating Window Attention (Constant VRAM):
Step 1: [ Window A ]
Step 2:      [ Window B ]
Step 3:           [ Window C ]
(Overlapping regions pass context across denoising steps to ensure global cohesion)

For ultra-high-definition frame synthesis, Herrmann et al. applied a parallel logic to video frame interpolation in High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion. Their model, HiFI, operates on overlapping image patches rather than full-resolution 4K or 8K images. By reusing a single, weight-shared diffusion model across all resolution scales and blending the patch boundaries, the framework handles massive spatial dimensions and large motions without triggering out-of-memory errors.

Sub-Quadratic Attention and Guidance

While cascades and windowing alter where and when computation occurs, another line of research targets the mathematical core of the memory bottleneck: the quadratic scaling of the attention mechanism itself.

To bypass this limitation and enable long-form, high-resolution generation, Wang et al. developed LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity. By replacing standard self-attention layers with a hybrid architecture called MATE, they achieved linear computational complexity relative to token length. This architectural shift allows a single GPU to synthesize minute-long, high-resolution videos without relying on external frame-interpolation pipelines.

Alternatively, instead of changing the attention mathematics entirely, one can use a low-resolution "proxy" to guide a sparse high-resolution attention mechanism. This is the intuition behind DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance.

In this work, Shen et al. demonstrated that a model can first compute a lightweight, low-resolution "draft" attention map. This draft acts as a sensor, identifying which spatiotemporal regions contain critical motion or structural changes. The model then uses this draft map to guide a hardware-optimized sparse attention mechanism at full resolution, computing attention only where it matters most. This training-free approach significantly cuts down memory overhead and accelerates inference while preserving fine visual details.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 345 papers
  2. Version 1 · 2026-10-04 · 430 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_high_resolution_video_synthesis_cacc7168,
  title = {high-resolution video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-resolution-video-synthesis/rdr_cacc7168a8644101b9460717cdbc7499},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_high_resolution_video_synthesis_cacc7168_v2,
  title = {high-resolution video synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-resolution-video-synthesis/rdr_cacc7168a8644101b9460717cdbc7499/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake