Hierarchical Coarse to Fine Video Synthesis

This direction is split into other directions: hierarchical video synthesis. Its last version stays readable and citable here.

Version 1 of 1130 papers158 conceptsfrozen 2026-10-04

Generating high-resolution, long-duration video is a computational nightmare. A single minute of 1080p video at 30 frames per second contains billions of pixels, all of which must remain spatially sharp and temporally consistent. If a model attempts to generate all these pixels simultaneously, it faces a "curse of dimensionality" that leads to massive VRAM requirements and "temporal drift," where the subject's identity or the scene's physics dissolve over time.

To solve this, researchers have converged on a shared strategy: Hierarchical Coarse-to-Fine Synthesis. Instead of generating the final video in one pass, these systems decouple the generation of global motion from the synthesis of fine-grained visual details.

The Global Blueprint

The first stage of a hierarchical pipeline focuses on the "storyline." By working at a lower resolution or a lower frame rate, the model can "see" the entire sequence's structure without being distracted by textures or lighting.

In ARLON, Shujie LIU and colleagues use an autoregressive model as a "coarse planner." This planner predicts discrete visual tokens that define the global motion and layout, providing a long-range temporal prior. Similarly, TempoMaster by Yue Ma generates a low-fps "global blueprint" first. This blueprint establishes high-level temporal semantics, ensuring that the beginning and end of a motion sequence are logically connected before any high-frequency details are added.

This stage effectively acts as a "sketch" of the video. As shown in ReVision by Qihao Liu, generating this initial draft at a lower resolution significantly reduces the computational bottleneck, as the model only needs to capture the "structural sketch" of the motion.

Anchoring and Interpolation

Once a global blueprint exists, the system must fill in the gaps. This is often done by treating the coarse frames as "anchors" or "keyframes."

LVDM by Qifeng Chen demonstrates this by first producing a sparse set of keyframes to establish the storyline, followed by a secondary diffusion model that performs temporal interpolation. This ensures that the intermediate frames don't wander off-script; they are mathematically constrained to transition smoothly between the established anchors.

ControlVideo, developed by Yabo Zhang, uses a similar hierarchical sampling strategy to manage memory. By generating sparse keyframes with global attention and then using them as anchors for intermediate frames, the model can produce long sequences on consumer-grade hardware that would otherwise be impossible to process.

High-Fidelity Refinement

The final stage is spatial and temporal refinement. Here, specialized "super-resolution" or "refiner" models take the low-resolution, temporally consistent video and inject high-frequency textures.

HunyuanVideo 1.5 utilizes a dedicated super-resolution stage to upscale its base model's output to professional-grade 1080p. This separation of concerns allows the base model to focus on motion logic while the refiner focuses on visual fidelity. InfinityHuman by Yi Ren applies this to human animation, using a pose-guided refiner to prevent "identity drift" and distorted limbs—common artifacts when a single model tries to handle both motion and anatomy at high resolution.

Logarithmic Scaling and Parallelism

Perhaps the most powerful intuition behind the hierarchical approach is that it transforms a linear problem into a logarithmic one. In traditional autoregressive generation, each frame depends on the one before it, creating a sequential bottleneck.

NUWA-XL by Shengming Yin introduces "Diffusion over Diffusion," a nested hierarchy where a global model generates coarse keyframes and local models recursively fill the gaps. Because the gaps are independent once the keyframes are set, they can be generated in parallel. This shift allows the total frame count to grow exponentially with each hierarchical level, leading to the 94% speedup in inference time reported by the authors.

By breaking the video into a "spacetime pyramid," as described in InfinityStar by Jinlai Liu, researchers have moved from struggling with 16-frame clips to generating minutes of coherent, high-definition video. The core lesson of this research direction is clear: to master the complexity of video, one must first master its structure at the coarsest scale.

Papers in this version

Showing 60 of 130 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_coarse_to_fine_video_synthe_7f80b3d7,
  title = {Hierarchical Coarse to Fine Video Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-coarse-to-fine-video-synthesis/rdr_7f80b3d7b3054b9395c3c16c332d97c9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_coarse_to_fine_video_synthe_7f80b3d7_v1,
  title = {Hierarchical Coarse to Fine Video Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-coarse-to-fine-video-synthesis/rdr_7f80b3d7b3054b9395c3c16c332d97c9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}