Coarse to Fine Generative Refinement

1,259 concepts831 papers9 proposals

When humans draw, we rarely start at the top-left corner and paint pixel-by-pixel until we reach the bottom-right. Instead, we sketch a loose, low-resolution layout to establish composition and proportions before filling in fine-grained textures. Yet, early autoregressive image models attempted exactly this "raster-scan" approach, treating images like one-dimensional text strings. This structural mismatch with 2D visual data often led to severe error accumulation, computational bottlenecks, and global incoherence.

The core intuition behind coarse-to-fine generative refinement is the separation of concerns. Modeling a high-resolution image all at once forces a network to simultaneously learn global semantic layouts (where a structural boundary lies) and high-frequency details (the exact texture of a surface). By decomposing the generation process into a multi-scale hierarchy, models can solve these problems sequentially: establishing global structural coherence at lower resolutions before refining local details at higher resolutions.

Breaking the Raster-Scan Bottleneck via Next-Scale Prediction

To bring the power of large language models to computer vision without the pitfalls of raster-scanning, recent architectures have shifted to a "next-scale" prediction paradigm. As demonstrated by Keyu Tian et al. in Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, a model can generate entire feature maps at progressively higher resolutions rather than individual tokens sequentially.

[Coarse Layout: 8x8] ---> [Intermediate Structure: 16x16] ---> [Fine Detail: 32x32]
       |                                 |                               |
       v                                 v                               v
Global Composition               Anatomical Logic                Texture & Edges

This scale-wise progression allows the network to lock in global structural consistency at lower resolutions first. Sucheng Ren et al. expand on this in M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation, showing that decoupling these scales allows the model to establish a robust global layout before spending computational cycles on local details. This drastically accelerates inference while maintaining high reconstruction fidelity, bypassing the quadratic complexity bottlenecks of standard token-by-token generation.

The Multi-Scale Pyramid and Residual Refinement

Why is this multi-stage approach so effective at generating sharp details? The answer lies in residual modeling. In a multi-scale generative pyramid, such as the framework proposed in SinGAN: Learning a Generative Model From a Single Natural Image, each successive layer does not need to reinvent the image. Instead, it only learns to produce a residual image—adding the missing high-frequency details to the upsampled output of the previous, coarser scale.

This additive approach stabilizes training significantly. Because global composition is determined almost entirely at the coarsest scale, users can even inject low-resolution manual sketches at the base level to guide the entire generation, a technique leveraged by frameworks like TOAD-GAN: Coherent Style Level Generation from a Single Example.

Furthermore, independent decoding of these hierarchical scales enables efficient inference strategies. In Enhancing Hierarchical Vector Quantized Autoencoders for Image Synthesis Through Multiple Decoders, Dario Serez et al. introduce a multi-decoder approach that creates a "preview mechanism." If the coarse structural representation generated at an early stage is determined to be low quality, the system can trigger an early exit, discarding the sample before wasting computational resources on high-resolution synthesis.

Hierarchical Trajectories in Continuous Generative Processes

The coarse-to-fine phenomenon is not exclusive to discrete token prediction; it emerges naturally in continuous generative models like diffusion and normalizing flows. In Variational Control for Guidance in Diffusion Models, Kushagra Pandey et al. observe that optimal control signals in diffusion processes inherently follow a hierarchical structure. The early timesteps of the reverse diffusion process establish global geometry and macro-features, while the final timesteps refine fine-grained textures and edges.

This temporal separation allows for massive scale jumps that would otherwise be computationally intractable. For instance, in ultra-resolution applications like histopathology, Sarah Cechnicka et al. develop URCDM: Ultra-Resolution Image Synthesis in Histopathology, chaining diffusion models in a hierarchical cascade. By using low-resolution global thumbnails to condition subsequent upsampling stages, the system maintains anatomical consistency from bird's-eye tissue architecture down to individual cell morphology.

+-----------------------------------------------------------------+
| Continuous Diffusion Trajectory                                 |
|                                                                 |
| Early Timesteps -------------------------> Late Timesteps       |
| (Global Geometry & Shape)                  (Local Textures)     |
+-----------------------------------------------------------------+
| Hybrid VAE-EBM Architecture                                     |
|                                                                 |
| Variational Generator -------------------> Energy Function      |
| (Coarse-Grained Blueprint)                 (Sharp Refinement)   |
+-----------------------------------------------------------------+

Similarly, hybrid models exploit this division of labor by pairing distinct generative paradigms. In VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models, Zhisheng Xiao pairs a coarse-grained variational generator with a fine-grained energy function. The variational component captures the global structure of the data distribution, leaving the energy-based model free to focus exclusively on refining sharp local details rather than modeling the entire data distribution from scratch.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 835 papers
  2. Version 1 · 2026-10-04 · 813 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_image_synthesis_cfc578a6,
  title = {hierarchical image synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-image-synthesis/rdr_cfc578a665704773b0b22413e7034f4a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_hierarchical_image_synthesis_cfc578a6_v2,
  title = {hierarchical image synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-image-synthesis/rdr_cfc578a665704773b0b22413e7034f4a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake