Teaching Fast Text-to-Image Samplers to Keep Words Attached to Regions

Modern image generators make a picture by starting from random noise and refining it over many small steps. Distillation methods compress those dozens of steps into one to four so that generation becomes interactive. The fast models mostly keep image quality, but they appear to lose something harder to measure: which word in the prompt controls which part of the image. When a prompt asks for "a red cube left of a blue sphere," a slow sampler gets many small steps to settle the layout and bind colors to objects. A one-step student has to make all of those decisions in a single network pass. The literature hints that this is where fast models break. Self-Corrected Flow Distillation reports sharp one-step images whose prompt adherence remains limited. Adaptive Matching Distillation explicitly favors image realism over prompt alignment when its few-step student drifts off the data manifold.

We propose to supervise the fast model's grounding directly. Grounding here means its cross-attention maps, which are the per-word heat maps that say where in the image each prompt token is being read. The targets come from the same network running in its slow, many-step mode. No external teacher, feature extractor, or reward model is involved. The new ingredient is the shape of the target. A one-step student must commit to layout at once, so we ask it to reproduce the teacher's attention maps aggregated over the stretch of the slow trajectory where layout is actually decided. Matching step-for-step is not an option because the student has no intermediate steps. This is a small change to an existing distillation pipeline, and it makes a sharp, falsifiable prediction. Few-step students should close most of the gap to their slow teacher on GenEval's position and attribute-binding categories without losing fidelity.

Core Claim

We make two claims. The first is that trajectory compression causes a specific, measurable loss: text-to-region binding degrades faster than overall image quality as the step count falls. The second is that this loss can be repaired with the model's own slow-mode cross-attention as a self-distillation target. The repair costs almost nothing in compute, because the slow-mode forward passes are already computed during distillation.

The claim is narrower than "fix compositional generation." GenEval shows that even strong slow models score only about 15% on spatial position and about 35% on attribute binding. We do not expect to beat the teacher. We expect to stop the student from falling below it.

Where This Sits

Three lines of prior work frame the proposal, and each plays a different role.

The first line motivates it. Work on cross-modal grounding shows that squeezing visual content into a global signal makes a model ignore the fine correspondence between words and regions. A visual attention grounding model for multimodal translation shows this for decoders, and MAF argues that token-to-region alignment is the remedy. Learning Robust Visual-Semantic Embeddings adds that instance-level matching aligns anchor points without aligning structure. We read few-step distillation as one more compression that reopens this word-to-region gap. Its output-matching losses are instance-level in exactly this sense.

The second line supplies the method. Mean Flow Distillation (MFD) distills a flow-matching model into a one- to four-step sampler by matching average velocities over time intervals. A flow-matching model is a generator that learns a velocity field carrying noise to data. MFD adds a small auxiliary flow model to prevent collapse to the dataset mean, and it releases code for SANA-1.6B as a LoRA student of the same network (happyw1nd/MFD). Because MFD's student is an adapter on the teacher's own weights, its output-level objective is already a form of self-distillation. Our addition is an internal target on top of it. SANA conditions on text through cross-attention layers, which gives us explicit per-token maps to supervise. KOALA supplies the closest precedent for supervising attention. It found that distilling self-attention maps from SDXL into a smaller U-Net prevents attributes from leaking between concepts better than output or feature distillation does. KOALA uses an external teacher, distills self-attention only, and compresses model size rather than step count. We distill cross-attention, use the same network as teacher, and target step compression.

The third line constrains the design. Internal self-distillation work warns that matching the deepest layers to the shallowest is counterproductive. The guidance has to go to middle layers, and prolonged matching chokes capacity, as seen in Self-Transcendence and Self-Regulation for Semantic Segmentation. Self-Transcendence also shows that a diffusion transformer's own internal features can beat external DINO guidance (7.51 versus 7.90 FID on ImageNet 256). It does this for training convergence of the full-step model, not for few-step sampling. Any-OPD is the closest work bridging representations during few-step distillation. It compares teacher and student outputs through a frozen DINOv2 global token. That signal is external, image-only, and global, so it cannot represent which word went where. OmniForcing handles a cross-modal gap during real-time distillation, but the gap it handles is temporal token-rate mismatch between audio and video in a causal generator. That is a different problem from text-to-region binding in parallel image sampling.

Research Program

The first stage measures the decay before building any fix. We take SANA-1.6B and run it in three ways: its full-step sampler, the same sampler truncated to 1, 2 and 4 steps without retraining, and MFD students at 1, 2 and 4 steps. We score every configuration on GenEval's per-category breakdown and on DPG-Bench. Alongside these scores we record two diagnostics of the student's cross-attention. One is per-token attention entropy. The other is the fraction of a noun or attribute token's attention mass that falls inside the box GenEval's detector assigns to that object. If the MFD students match the teacher on position and color attribution within noise, the idea is unnecessary and we stop.

The second stage builds the grounding target. During MFD training the frozen teacher is already integrated across each interval $[s,t]$ to form its average velocity, so its cross-attention maps can be captured at no extra forward cost. Let $A^{\theta}_{l,k}(x,t,c)$ be the normalized spatial cross-attention distribution of text token $k$ at layer $l$, for latent $x$ at noise level $t$ and prompt $c$. The teacher is the frozen base model $\theta_0$; the student is the adapted model $\theta$. We form the teacher target by averaging over the teacher steps $t_i$ that fall in a layout window $W$ of noise levels. We locate this window empirically in stage one as the range where teacher attention maps stop changing their argmax region. We then add a grounding term to MFD's loss:

$$\mathcal{L}_{\text{ground}} = \lambda(\tau) \sum_{l \in \mathcal{M}} \sum_{k \in \mathcal{K}(c)} \mathrm{JS}\Big( A^{\theta}_{l,k}(x_{t_s}, t_s, c) \;\Big\|\; \tfrac{1}{|W|}\textstyle\sum_{t_i \in W} A^{\theta_0}_{l,k}(x_{t_i}, t_i, c) \Big).$$

The symbols are as follows. $t_s$ is the noise level at which the student is queried, which is the initial noise level in one-step sampling. $\mathcal{M}$ is a set of middle cross-attention layers, chosen in line with the warnings against extreme-layer matching. $\mathcal{K}(c)$ is the set of noun and attribute tokens in the prompt, found by a part-of-speech tagger. $\mathrm{JS}$ is the Jensen-Shannon divergence, chosen because it is bounded when maps are sharp. $\lambda(\tau)$ is a weight that decays with training step $\tau$, which follows the same internal-distillation warnings against over-regularization.

The third stage asks why the fix works, and it rests on ablations that isolate each ingredient. One replaces the trajectory-aggregated target with the teacher map at the single matched noise level. Another substitutes self-attention for cross-attention, which is KOALA's choice. A third matches raw hidden features at the same layers instead of attention. A fourth uses a target from a mismatched prompt, so we can tell whether the gain comes from grounding information or merely from regularizing attention.

Baselines

The naive reference point is SANA-1.6B's own sampler truncated to the target step count with no retraining, which shows how much grounding a solver loses by itself. The full-step teacher is reported as the ceiling.

The strongest existing method on this exact question is MFD applied to SANA-1.6B. It shares our backbone, our code, and our self-teaching setup, so any difference isolates the grounding term. The MFD paper reports an aesthetic score of 6.541 and a CLIP score of 29.430 at 4 steps on LAION-aesthetic-6.5+. It reports no GenEval or DPG-Bench result, so no reported result was found on the file for MFD on GenEval. We will rerun it with the released code at the reported hyperparameters and match training images, GPU-hours, and learning-rate sweep size between MFD and MFD plus grounding. Beating it means a statistically reliable gain on GenEval position and color attribution, over three seeds, at 1 and 2 steps. That gain must come with COCO FID and MFD-style recall within one seed-standard-deviation of MFD.

For external context we cite Adaptive Matching Distillation's GenEval overall of 0.57 on SDXL, and Any-OPD's DrawBench PickScore of 0.884 against its 0.878 teacher. Both use different backbones, so they are orientation rather than head-to-head numbers.

Two further baselines represent the external-signal routes. One is an Any-OPD-style bridge, in which we add a frozen DINOv2 global-token loss to MFD in place of our attention term. The other is a reward fine-tuning stage on the MFD student. For this we use the detached-target recipe of DiffusionOPSD with PickScore_v1 as reward. Both receive the same extra GPU-hours that our term consumes.

The ablations in stage three serve as the internal controls. The mismatched-prompt target in particular must perform no better than plain MFD for our explanation to hold.

Why not just…?

Why not just train MFD longer, since output matching should already carry grounding? This is the main threat. If the slow teacher's images contain correct layouts, a perfect output match implies correct grounding. Our bet is that, at a fixed budget, the average-velocity objective spends its capacity on texture and global statistics before it captures binding. Average velocity is a pixel-weighted signal in which a swapped color on a small object is a tiny error. Stage one tests this directly by tracking GenEval binding against training length for plain MFD. If the gap closes with more MFD training, we report that and drop the method.

Why not just fine-tune the fast model with a reward model? Reward models are trained on preference, not on binding. Reward fine-tuning also tends to be expensive, and DiffusionOPSD reports 40 to 63% fewer GPU-hours than unrolled reward methods. It also observes a reversal in which higher-gradient targets yield less gain. AMD shows that reward signals become unreliable once a student drifts off-manifold. These cautions do not settle the matter, so we run reward fine-tuning at matched compute and compare it per GenEval category.

Why not just use DINOv2 or CLIP as the bridge, as Any-OPD does? A global image token cannot encode which word went to which region, and Any-OPD itself notes the bridge is image-specific. CLIP-style scores track GenEval's object checks only moderately: the GenEval authors report 80% human agreement for CLIPScore against 83% for GenEval. The DINOv2-bridge baseline will show whether a global external signal recovers binding. We predict it improves fidelity but not position or attribution.

Why not just steer attention at inference time? Test-time attention optimization spends extra gradient steps per image, which defeats the latency goal. We report its cost in network evaluations if it is included as a reference. The comparison we care about is at equal sampling cost.

Proposed Experiments and Decisive Checks

All training uses SANA-1.6B with LoRA through the MFD codebase on LAION-aesthetic-6.5+, at 1, 2 and 4 sampling steps. Prompt adherence is evaluated with GenEval per category, DPG-Bench, and DrawBench PickScore. Fidelity and diversity are evaluated with MS-COCO FID and CLIP score and with recall in MFD's protocol. The attention-to-box agreement diagnostic is computed with GenEval's detector boxes.

The first decisive check is the existence of decay. MFD students must lose at least several GenEval points on position or color attribution relative to the full-step teacher, while losing little FID. Otherwise the problem is not real.

The second check is that the grounding term recovers a majority of that per-category loss at 1 and 2 steps, with no recall drop. The mismatched-prompt ablation must recover none of it.

The third check concerns mechanism. If trajectory-aggregated targets beat matched-noise-level targets specifically at one step, that supports our reading: the student needs the teacher's settled layout, not its noisy early guess. Cross-attention must also beat self-attention and hidden-feature matching on binding, which separates our signal from KOALA's.

Finally, we report the added training cost of the grounding term, expected to be small because teacher passes are reused. We also report inference cost, which by construction is unchanged.