Coarse-to-Fine Semantic Scaffolding for Image Generation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2813 papers999 conceptsfrozen 2026-10-04

Traditional image generation models often struggle with a fundamental trade-off: maintaining a coherent global structure while synthesizing high-frequency local details. When a model attempts to generate a high-resolution image in a single pass or via a simple raster-scan (pixel-by-pixel) sequence, it often loses the "big picture," leading to artifacts like repeated limbs, incoherent textures, or "Frankenstein" compositions.

A growing body of research suggests that the solution lies in coarse-to-fine semantic scaffolding. Instead of treating all pixels as equal, these models decompose generation into a hierarchy: first establishing a low-resolution "blueprint" of the scene’s layout and then iteratively refining that scaffold into a high-fidelity image.

The Blueprint Intuition

The core intuition behind this direction is that visual information is naturally hierarchical. A human artist does not paint a masterpiece by finishing one square inch at a time; they start with a rough sketch to fix the composition.

In the realm of autoregressive models, Keyu Tian and colleagues introduced Visual Autoregressive (VAR) modeling (Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction), which replaces the traditional raster-scan order with "next-scale prediction." By predicting the image as a series of increasingly dense token maps, the model ensures that the global layout is locked in before the first fine detail is even considered. This approach mirrors the Spectral Image Tokenizer (SIT) developed by Carlos Esteves and others (Spectral Image Tokenizer), which uses wavelet transforms to generate a blurry "big picture" first.

This hierarchical progression is not just an aesthetic choice; it is where the "intelligence" of the model resides. Kai Wang demonstrates in DiverseVAR that the fundamental diversity and composition of an image are determined almost entirely during these initial coarse scales.

Decoupling Structure from Detail

By separating the "what goes where" (structure) from the "what it looks like" (detail), researchers can optimize each stage independently. This decoupling is the secret to the efficiency of models like Generation then Reconstruction (GtR) by Feihong Yan (Generation then Reconstruction: Accelerating Masked Autoregressive Models via Two-Stage Sampling). GtR uses a slow, careful process for the "semantic scaffolding" and a lightning-fast, parallel process for "detail reconstruction," achieving significant speedups without losing quality.

Similarly, Senmao Li and the authors of StageVAR argue that because the global plan is fixed early, the final refinement stages can use much lighter computational mechanisms. This temporal shift—from semantic definition to fidelity refinement—is also observed in diffusion models. Qiucheng Wu showed that explicitly controlling spatial-temporal attention to prioritize global structure in early denoising steps significantly improves compositional fidelity (Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis).

Scaling to Gigapixels

The scaffolding approach is particularly powerful for generating extremely high-resolution images (4K and beyond) without the need for massive retraining.

  1. Low-Res Previews: Models like DemoFusion (Ruoyi Du) and ZoomLDM (Srikar Yellapragada) generate a low-resolution base image that acts as a global constraint.
  2. Patch-wise Refinement: Once the global structure is established, the model can denoise individual high-resolution patches. Because these patches are all conditioned on the same low-res "blueprint," they remain contextually coherent.
  3. Self-Cascading: Lanqing Guo and others (Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation) have even shown that a single model can be recycled as its own super-resolution stage, using its internal features at one scale to guide the next.

This "blueprinting" phase provides the spatial context needed to coordinate high-frequency details across independent patches, solving the "object repetition" problem that plagues naive high-res extrapolation (AccDiffusion by Mingbao Lin).

Hierarchical Guidance

Finally, this research direction rethinks how we interact with models. If the model generates images hierarchically, our instructions should be hierarchical too.

SCoPE, developed by Ketan Suhaas Saichandran, breaks complex prompts into a sequence of increasingly granular sub-prompts (Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Models). By feeding the model coarse descriptions first and saving the intricate attributes for later, SCoPE ensures the model doesn't get "distracted" by fine details before the scene layout is stable. This is echoed in HiGFA (Zhiguang Lu), which sequences different guidance signals—starting with text for layout and transitioning to fine-grained classifier signals for category-specific details.

By aligning the generation process with the natural hierarchy of visual information, these models achieve a level of structural consistency and scaling efficiency that "flat" models cannot match.

Papers in this version

Showing 60 of 813 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 835 papers
  2. Version 1 · 2026-10-04 · 813 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_image_synthesis_cfc578a6,
  title = {hierarchical image synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-image-synthesis/rdr_cfc578a665704773b0b22413e7034f4a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_coarse_to_fine_semantic_scaffolding_for__cfc578a6_v1,
  title = {Coarse-to-Fine Semantic Scaffolding for Image Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/coarse-to-fine-semantic-scaffolding-for-image-generation/rdr_cfc578a665704773b0b22413e7034f4a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}