When you ask a modern text-to-image diffusion model to generate "a photo of a cat," the results are often breathtaking. But if you ask for "a blue cube on top of a red plate to the left of a yellow coffee mug," the system frequently unravels. It might omit the cube entirely, paint the mug red, or scramble the spatial layout.
This breakdown is known as compositional misalignment. It manifests in two primary ways: catastrophic neglect (where requested objects are completely omitted) and spatial reasoning failure (where objects are rendered but placed in positions that contradict the prompt).
To understand why this happens—and how a recent wave of research fixes it—we have to look under the hood at how diffusion models translate words into pixels.
Why Compositionality Fails: The Attention Bottleneck
Diffusion models generate images iteratively, starting from pure Gaussian noise and gradually shaping it over dozens of timesteps. This process is guided by the text prompt via cross-attention mechanisms. At each step, visual features in the image latent space query the textual embeddings of the prompt to determine what features to surface and where.
This is where the bottleneck occurs. In complex, multi-object prompts, the cross-attention maps for dominant subjects or global textures can easily overpower weaker tokens. If an object token fails to secure enough "attention real estate" during the critical early phases of denoising, it is permanently washed away by the noise-removal process.
Furthermore, standard text encoders (like CLIP) treat prompts more like a "bag of words" than a structured geometric blueprint. They excel at capturing high-level semantic concepts but struggle with precise relational logic, such as attribute binding (linking "blue" specifically to "cube") and spatial positioning ("to the left of").
To solve this, researchers have developed four complementary strategies: structured visual planning, test-time attention guidance, early-detection resampling, and alignment fine-tuning.
Strategy 1: Structured Visual Planning
If text encoders struggle with spatial logic, why not decouple the planning of the scene from the generation of the pixels?
A prominent line of work introduces an intermediate representation—such as a layout or a structured data object—between the text prompt and the diffusion model. Instead of forcing the diffusion model to implicitly deduce coordinates from natural language, an external planner explicitly maps out the scene first.
[Text Prompt] ──> [Structured Planner (LLM/JSON)] ──> [Layout/Bounding Boxes] ──> [Diffusion Model] ──> [Aligned Image]
For instance, Weixi Feng et al. developed LayoutGPT, which leverages the latent coding knowledge of Large Language Models (LLMs) to generate 2D or 3D layouts formatted as CSS code. By translating spatial instructions into structured code, the LLM acts as a visual planner that specifies exactly where each object count and boundary should sit.
Similarly, Yuanhuiyi Lyu et al. introduced StruVis, a framework that trains models to "think" in structured visual data (like JSON) before generating raw pixels. This ensures that attribute binding and spatial relationships are locked into a structured intermediate state. Taking this concept into a reinforcement learning framework, Mingrui Wu et al. developed RePrompt, which trains an LLM to reason about a scene's layout and generate an enhanced, spatially explicit prompt before the image generator ever begins its work.
Strategy 2: Dynamic Attention and Embedding Guidance
What if we want to fix compositional alignment without adding an explicit planning step or retraining the base model? We can intervene directly within the cross-attention layers during the denoising process.
If certain tokens are being neglected, we can dynamically manipulate their attention weights to force the model to render them. Zhiyuan Chang et al. demonstrated this with Patcher, an automated tool that identifies missing objects in complex prompts and injects explicit physical details or specific terminology to rebalance the model's internal attention maps.
Standard Denoising: [Token: "cube"] ── (Low Attention) ──> Omitted from Image
With Guidance/FRAP: [Token: "cube"] ── (Boosted Weight) ──> Rendered in Image
Rather than modifying the text, Geon Yeong Park et al. proposed Energy-Based Cross Attention (EBCA)](/work/energy-based-cross-attention-for-bayesian-context-update). This method treats cross-attention as an energy minimization problem, adaptively updating the text embeddings at each generation step to ensure that every concept in the prompt maintains a strong visual footprint. Similarly, Liyao Jiang et al. introduced FRAP, which utilizes adaptive prompt weighting during the generation process to dynamically scale the importance of individual words, preventing dominant subjects from completely erasing secondary entities.
To address part-level omissions (such as missing sleeves or pockets in fashion generation), Xujie Zhang et al. developed DiffCloth, which explicitly aligns linguistic "Attribute-Phrases" with specific visual sub-regions to eliminate part leakage and color bleeding.
Strategy 3: Test-Time Resampling and Early Detection
Because diffusion generation is a stochastic process governed by an initial random seed, some generation trajectories naturally lead to failures while others succeed. Instead of blindly running a bad trajectory to completion, we can monitor the latent space and course-correct early.
Yujian Liu et al. tackled this by introducing a particle filtering framework in Correcting Diffusion Generation through Resampling. By utilizing external guidance—like object detectors—to evaluate intermediate latents during denoising, the framework resamples particles that are drifting toward object omission, steering the trajectory back toward semantic faithfulness.
Denoising Steps: [ t=50 ] ──> [ t=40 ] ──> [ t=30 (Check: Is object forming?) ]
│
├──> No ──> [ Resample / Restart ]
└──> Yes ──> [ Continue to t=0 ]
To save computational resources, Lorenzo Baraldi 0002 et al. proposed HEaD+](/work/hallucination-early-detection-in-diffusion-models), a framework designed for the early detection of hallucinations and omissions. If the system detects within the first few denoising steps that a requested object will fail to materialize, it immediately aborts the run and restarts with a different seed, preventing the waste of downstream compute.
Strategy 4: Alignment Fine-Tuning and Reinforcement Learning
A final approach seeks to permanently fix the underlying model's behavior through targeted training strategies, teaching it to inherently respect spatial and compositional boundaries.
One intuitive way to do this is by learning from mistakes. Xinyan Chen et al. introduced Iterative Prompt Relabeling (IPR)](/work/iterative-prompt-relabeling-for-diffusion-model-with-rldf). When a model generates an image that mixes up spatial positions or omits an object, an object detector "relabels" the image with a description of what was actually rendered. The model is then fine-tuned on these corrected pairs, turning its failures into accurate training data.
To optimize models directly for human spatial perception, Parham Rezaei et al. moved away from rigid center-point distance checks, proposing a Probability of Superiority (PoS) approach called PSG to evaluate and guide spatial relationship alignment more fluidly.
Traditional Reward: Did the object land exactly on coordinate X? (Sparse/Rigid)
PoS / PSG Reward: Is object A visibly to the left of object B? (Continuous/Human-Aligned)
Training via Reinforcement Learning (RL) is highly effective for this task, but it suffers from "sparse rewards"—the model only receives feedback after the final image is generated, making it difficult to know which specific denoising step caused an object to disappear. To solve this, Zijing Hu et al. developed $B^2$-DiffuRL](/work/towards-better-alignment-training-diffusion-models), which uses backward progressive training and branch-based sampling to isolate exactly which denoising actions successfully preserve attribute binding and spatial relationships.
Finally, for a lightweight alternative to full model retraining, Jaa-Yeon Lee et al. introduced SoftREPA](/work/aligning-text-to-image-in-diffusion-models). This strategy keeps the base diffusion model entirely frozen and uses contrastive learning to optimize a tiny set of "soft text tokens" (under 1 million parameters), explicitly teaching the text-conditioning interface to distinguish between well-aligned and poorly-aligned text-image pairs.
Key Foundational Work
Learning from Mistakes: Iterative Prompt Relabeling for Text-to-Image Diffusion Model Training (2024) — contributed 1 concept elements to this direction.
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023) — contributed 1 concept elements to this direction.
Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement (2024) — contributed 1 concept elements to this direction.