Why Flat Drafts Stay Flat: Region-Mean Latents Govern Texture in Flow-Based Image Generation

Zihan Gu, Jiangcan Jia, Hua Zhang, Lulu ZhaoarXiv 2026 · arXiv

TL;DR

This paper uncovers a surprising link in flow-based image generators: the "flatness" of images generated from simple color-block drafts is governed by coarse regional means in the VAE latent space, not just missing high-frequency detail. By correcting these regional latent means—either through training-free pixel dithering or guided LoRA adaptation—the authors enable photorealistic texture generation while strictly adhering to the draft's layout and color.

What Problem Does This Paper Address?

When researchers and artists use "flat" color-block drafts to guide image generation (a common technique for controlling layout and color), they often hit a wall: the resulting images stay stubbornly flat. Even when the prompt explicitly asks for a "photorealistic RAW photo," the generator produces smooth, plastic-looking surfaces that mirror the draft's lack of detail.

This "persistent flatness" is a major hurdle for image-to-image workflows. Standard techniques like SDEdit (adding noise to the input) face a difficult trade-off: moderate noise preserves the draft's layout but keeps the surfaces flat, while heavy noise allows for texture but destroys the intended regional colors and boundaries. The authors set out to find the root cause of this behavior in modern flow-based models like FLUX.1 and Qwen-Image.

Methods

The authors trace the problem back to the Variational Autoencoder (VAE) used by these models. They discover that the process of "flattening" an image systematically shifts its regional latent means along a specific channel direction.

To address this, they propose several interventions:

  • Mechanism Analysis via Field Exchange: They independently swap the "regional mean fields" and "complementary fields" (the high-frequency residuals) between paired drafts and photos. This confirms that the mean field alone acts as the "gatekeeper" for texture.
  • Training-Free Correction (Dither & Hold): They introduce low-amplitude pixel dither before VAE encoding. This exploits the encoder's non-linearity (curvature) to shift the latent means into a state that "releases" texture. To prevent this noise from blurring edges, they use "boundary holding," which resets tokens near region boundaries during the early steps of the generation process.
  • Paired Adaptation (LoRA): They train a Low-Rank Adapter (LoRA) on pairs of flat drafts and photos. They show that the standard "velocity objective" used in training naturally penalizes errors in regional latent means.
  • Explicit Mean Guidance: They introduce an inference-time guidance term that allows users to explicitly steer the regional means toward a target "photorealistic" level, providing a continuous dial for texture control.
Figure 2. Regional means connect encoding, trajectory retention, and rendering control. Pre-encoding dither produces a latent displacement; its mean field is scaled while its spatial complement is retained. The lanes represent the operational split P and I −P. Regional noise averaging motivates high mean retention. Boundary tokens are held early and then released; optional draft-conditioned LoRA and mean guidance act on the same velocity predictor. The guidance target uses δd = A(zd −zf). Real FLUX.1-dev panels show one saved case: the draft (left), dither with mean scale 1.15 and boundary holding (upper right), and the flat-start rendering (lower right). These panels illustrate the training-free branch. Paths, fields, and coordinate insets are schematic; the gray path denotes a separate uncorrected run.

Datasets

The study utilizes several datasets for analysis and evaluation:

  • COCO val2017: Used to create "mechanism" and "confirmation" sets of 300 images each, featuring diverse natural scenes.
  • Synthetic Block Set: 5,000 stimuli consisting of base colors with various noise and texture perturbations to probe VAE encoder sensitivity.
  • Layered Set: 100 programmatic layouts used to measure how well the model retains regional information across different region sizes.
  • LoRA Training Set: A specific subset of 2,221 paired draft/photo images derived from COCO.

Key Insights and Ideas

  • Texture is a Coarse Statistic: Counter-intuitively, the presence of fine-grained texture in the final output is determined by the coarse-grained average value of latent vectors within a region. If the mean is "wrong," the generator refuses to synthesize detail, regardless of the prompt.
  • The Encoder's Curvature is Key: The VAE encoder isn't just a linear downsampler. Its mathematical "curvature" is what maps pixel-level variance (like dither) into the specific latent direction that signals the generator to create texture.
  • High Retention in Flow Models: Flow-based generators are remarkably sensitive to these regional means, retaining them with near-unit sensitivity. This explains why a "flat" latent starting point almost inevitably leads to a "flat" final image.
Figure 1. Flatness follows regional latent means. FLUX.1-dev with matched prompt/noise. Left: draft d and photo x, with equal regional RGB means before quantization. Their VAE latents zf, zp have different regional means, measured by A. In the output grid, rows select draft/photo mean fields and columns select draft/photo complementary fields (b = 0/1), using the soft-mask operator P. The photo complement alone leaves surfaces flat; the photo mean field restores texture. Matrix T: geometric texture ratios over 300 images. Curves: 100 images; dashed lines mark source texture T = 1 and identity π = α.

Key Results

The authors demonstrate significant improvements in both training-free and adapted scenarios:

  • Texture Restoration: In FLUX.1-dev, replacing a draft's mean field with a photo's mean field raises the geometric texture ratio from a flat 0.12 to a rich 1.31.
  • Quantitative Superiority: Their "Dither + Hold" method outperforms standard SDEdit and ControlNet baselines. On a 300-image benchmark, their LoRA + mean guidance approach achieved a human preference score (HPSv3) of 12.56, significantly higher than the 10.71 achieved by a tuned ControlNet Tile baseline.
  • Continuous Control: The "mean-field dose" ($\alpha$) acts as a reliable "texture dial." Users can smoothly transition from a flat vector-style illustration ($\alpha=0$) to a highly detailed photograph ($\alpha=1.15$ or higher) while maintaining perfect layout fidelity.
Varying the target scale $\alpha$ allows for continuous control over surface detail.

Limitations and Future Work

While the paper provides a robust explanation for persistent flatness, some challenges remain:

  • Semantic Ambiguity: Restoring texture doesn't always fix semantic errors. For example, a flat silhouette of a person might still be misinterpreted by the model as a piece of architecture, even if it now has a "brick" texture.
  • Empirical Direction: The "texture-releasing" direction in latent space is currently identified empirically for each model backbone. A more fundamental, theoretical derivation of this direction would be valuable.
  • Fixed Masks: The current implementation relies on fixed region masks. Future work could explore how to handle dynamic or soft boundaries more gracefully.

Summary updated

Abstract

Flat color-block drafts specify layout and regional color while leaving interior detail unspecified, yet pretrained flow-based generators can preserve their flat appearance even under photorealistic prompts. We trace this behavior to a cross-scale mechanism linking fine pixel variation to regional means in variational autoencoder (VAE) latent space. Flattening photo regions systematically shifts their latent means, with offsets concentrated along a shared channel direction within each backbone. Controlled field exchanges in FLUX.1-dev and Qwen-Image establish the effect of this displacement on texture generation: adding the photo-derived regional mean field to a flat latent restores texture, whereas adding its complementary field alone largely preserves flatness. A second-order encoding expansion explains how encoder curvature can convert zero-mean perturbations into mean shifts. An analytically solvable conditional Gaussian flow shows how regional noise averaging enables near-unit endpoint sensitivity at high signal-to-noise ratios. This mechanism motivates training-free rendering through pixel dither before encoding, mean-field scaling, and early boundary holding. Paired low-rank adaptation (LoRA) with draft conditioning improves content recovery and produces photo-like regional means under the standard velocity objective. Explicit regional mean guidance remains effective after adaptation, continuously controlling texture and improving color fidelity. These findings connect encoding shifts to persistent generation behavior and support interventions evaluated for texture, spatial fidelity, and semantic recovery.
BibTeX
@misc{gu2026why,
  author = {Zihan Gu and Jiangcan Jia and Hua Zhang and Lulu Zhao},
  title = {{Why Flat Drafts Stay Flat: Region-Mean Latents Govern Texture in Flow-Based Image Generation}},
  year = {2026},
  eprint = {2611.25930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25930},
}
Figure