Echo-Ordered Guidance: Pulse Sequences for Multi-Concept Diffusion Sampling

When a text-to-image model is asked for "a blue bench and a green car", it often drops one object or paints the bench green. Text alone does not tell the model which words belong to which region. The fixes that work best today nudge the image during generation so that each named object gets attention. When two such nudges pull the image in opposite directions, current methods either add them together, where they partly cancel, or push only the weakest object at each step. Adding fails when the pulls conflict. Pushing only the weakest object lets whichever object was pushed last win. We propose to schedule the nudges the way quantum physicists schedule control pulses in dynamical decoupling. Dynamical decoupling protects a fragile qubit (a quantum bit) from slow background noise by flipping it with short pulses in carefully ordered, symmetric sequences, applied only when they help.

The central claim is concrete and testable. Applying one concept's guidance at a time, in a palindromic order such as A, B, B, A, and only during the short stretch of sampling when the image commits to its layout, beats summed guidance at the same number of gradient evaluations. It should also remove the systematic bias toward whichever concept is guided last. Multi-objective optimization supplies both the competitor and the measuring stick. The main rival is a Pareto-style gradient projection, which finds a direction that improves every objective at once. Every method is judged by the trade-off frontier it reaches between object presence, attribute binding, image quality, and latency.

Core Claim

A diffusion model is a generator that starts from random noise and removes it step by step. Inference-time guidance changes the partly denoised latent at some of those steps by gradient descent on a loss computed from the model's cross-attention maps. A cross-attention map shows, for each word, which image regions are attending to it. For two concepts A and B with losses $L_A$ and $L_B$, a single guidance step on concept A is $G_A(z) = z - \eta \nabla L_A(z)$, where $z$ is the latent and $\eta$ the step size.

Expanding to second order in $\eta$ shows where the ordering matters:

$$G_B(G_A(z)) - G_A(G_B(z)) = \eta^2 \left( H_A \nabla L_B - H_B \nabla L_A \right) + O(\eta^3),$$

Here $H_A$ and $H_B$ are the Hessians (second-derivative matrices) of the two losses. The right-hand side is a commutator term: a systematic favoritism toward whichever concept is applied second. A palindromic sequence $G_A G_B G_B G_A$, with each step at half size, cancels this term at second order. This is the same reason symmetric echo sequences cancel leading-order errors in the quantum setting.

Summed guidance, $z - \eta(\nabla L_A + \nabla L_B)$, has no ordering bias. It does lose the step whenever $\langle \nabla L_A, \nabla L_B \rangle < 0$, that is, whenever the two gradients point in partly opposite directions. Alternation keeps the useful second-order term, in which B responds to where A has already moved the latent. Symmetric alternation keeps that benefit while removing the bias.

This yields three predictions. Gains over summing should concentrate on prompts where the two attention-loss gradients have negative cosine. Plain alternation (A, B, A, B) should show an order effect: swapping which concept is guided last should change which object survives. The palindromic order should shrink that effect.

Grounding in Prior Work

The proposal draws on each direction in a different role.

Dynamical decoupling supplies the method. Periodic and echo pulse sequences cancel slowly varying noise in flux qubits (Trappen et al.) and in solid-state spin baths (Rong et al.). Our symmetric orderings are borrowed directly from this sequence-design logic.

ADAPT supplies a second ingredient and a warning. It shows that inserting pulses everywhere is worse than inserting them selectively: it reports a 1.2x fidelity gain over decoupling on all qubits. It chooses where to pulse using cheap stand-in circuits ("decoy circuits") whose behavior can be simulated in advance. We reuse that idea in the form of a cheap probe that predicts which concept is at risk of being dropped. ADAPT works on discrete quantum circuits and has nothing to say about images.

Group-theoretic RL of decoupling sequences searches for sequences that cancel higher-order error terms under a single reward. We use only fixed, hand-designed sequences at first, and consider learned sequences only if the simple ones work.

The phrase "dynamical decoupling" in Montanari & Urbani refers to a separation of timescales during network training. It is a different sense of the term and plays no role here beyond the general intuition that structure and noise act at different times.

Prompt ambiguity motivates the work. Attend-and-Excite shows that objects are dropped within the early denoising steps. It fixes this by repeatedly maximizing the attention of the most neglected word, which roughly doubles sampling time (about 5.6s to 11.8s on an A100). Its greedy max-of-min rule is precisely the "push the weakest object at each step" policy whose ordering bias we target.

Composable Diffusion composes per-clause predictions by summation. It reports visual crowding when clauses compete for the same region, which is the cancellation failure we target. Rich Text separates attributes into region-specific losses but requires the user to supply the regions. We keep the plain prompt and act only on the schedule.

Theory constrains when to pulse. Dynamical regimes of diffusion models divides the reverse trajectory into three phases. It begins with pure noise, then a "speciation" window in which the sample commits to a coarse class, and finally a "collapse" phase that settles fine detail. The speciation time is set by $\Lambda e^{-2t_S}=1$, where $\Lambda$ is the largest eigenvalue of the data covariance.

That analysis covers unconditional sampling with an idealized noising process, so we treat the predicted window as a hypothesis and also measure the commitment point empirically. We should be clear that Attend-and-Excite already restricts guidance to early steps. What we add is a measured window and sparse pulses within it, not windowing as such.

Multi-objective optimization frames the evaluation. Linear weighting cannot reach non-convex parts of a trade-off front (Cheng et al., Zhang). We therefore compare methods by the fronts they trace, not by any single weighted score.

Research Program

We build a guidance scheduler on top of the Attend-and-Excite code path for Stable Diffusion. Each concept has its own loss: one minus that concept's smoothed maximum attention, as in Attend-and-Excite. The scheduler allots a fixed budget of $P$ backpropagations ("pulses"). It places them inside a window of denoising steps and assigns each pulse to a single concept according to a sequence: summed, max-of-min, A–B alternation, A–B–B–A palindrome, or random order. For three concepts we use cyclic palindromes such as A–B–C–C–B–A.

We locate the window in two ways. The first is the commitment step, defined as the step after which the argmax regions of the attention maps stop moving. The second is the theoretical speciation time, computed from the covariance of the latent dataset.

An ADAPT-style probe runs one cheap forward pass at a low-resolution early step. It ranks concepts by predicted neglect, and that ranking decides which concept opens the sequence and whether pulses are spent at all. Prompts whose concepts are already all present receive none. For every prompt we also log the gradient cosine between concepts, so that each gain can be attributed to conflicting or cooperating prompts.

Baselines

The do-nothing reference is plain Stable Diffusion with classifier-free guidance at the same sampler and step count. On the Attend-and-Excite prompts it reaches BLIP caption similarity of 0.76–0.79, against 0.80–0.83 for Attend-and-Excite as reported there.

Attend-and-Excite with its published schedule is the strongest existing inference-time method for neglect, and its numbers above are the bar on its own benchmark. We rerun it under our seeds rather than copy figures. Beating it means a higher score for the most neglected object and higher caption similarity at no more backpropagations, or matching it at clearly fewer.

Summed per-concept guidance, as in Composable Diffusion, is the linear-composition reference. We sweep a grid of weights for it so that it traces its best front, not a single point.

The strongest threat is a Pareto-style gradient projection. At each pulse it combines the concept gradients so that no concept's loss increases. It is the standard multi-objective remedy for gradients that point in opposite directions. We found no reported result on Lacuna for gradient projection applied to text-to-image attention guidance, so we implement it ourselves, at the same pulse budget and window.

Three ablations isolate the proposal's ingredients. Plain alternation against palindromic alternation isolates the symmetric ordering. Random order controls for alternation as such. A full-trajectory schedule against a windowed one isolates the timing.

For context on T2I-CompBench attribute binding, T2I-CompBench reports that GORS fine-tuning raises B-VQA (a question-answering score of attribute binding) from 0.5065 to 0.6603 on Stable Diffusion v2. That is a training-time method, reported only as an upper reference and not as a like-for-like competitor.

Every method shares prompts, seeds, sampler, step count, pulse budget, and an equal tuning budget of the same number of configurations evaluated on a held-out prompt split.

Why not just…?

Why not just project the gradients onto a common improving direction instead of alternating? This is the real competitor. If projection matches palindromic alternation on both frontier quality and order-swap asymmetry, the decoupling contribution reduces to known multi-objective practice. We will report that outcome as such.

Our expectation that alternation can win rests on the second-order term in the expansion above. Projection discards the conflicting component of each gradient. Alternation lets the second concept respond to the latent the first has already changed. The per-prompt gradient-cosine analysis settles the question: alternation's advantage, if real, should grow as the cosine becomes more negative.

Why not just tune the guidance weights? Weighted sums cannot reach non-convex parts of the front (Zhang). Summing also cancels exactly when the gradients conflict, which Composable Diffusion observes as crowding. The weight sweep is included, so the claim fails directly if its front matches ours.

Why not just sample many images and keep the best one under a binding scorer or PickScore? For prompts with several independent constraints, the chance that a random sample satisfies all of them is the product of the per-constraint chances, so the required number of samples grows quickly. Scalar reward selection can also exploit quirks of the scorer (Diffusion-LPO). This is still a plausible competitor at small numbers of objects. We run best-of-N at matched wall-clock time and report where its front crosses ours.

Why not just rewrite the prompt with a language model? T2I-CompBench++ finds that even models with strong internal prompt processing still fail at spatial relations and counting. Rewriting is also orthogonal to scheduling, so we test it in combination with our method rather than as a replacement.

Proposed Experiments

We evaluate on the Attend-and-Excite benchmark (animal–animal, animal–object, and object–object pairs) and on the color, shape, and texture splits of T2I-CompBench, using Stable Diffusion 1.4 and 2.1. Metrics are:

  • the CLIP score of the most neglected object and BLIP caption similarity, both from Attend-and-Excite;
  • B-VQA for attribute binding;
  • PickScore as an image-quality proxy;
  • wall-clock latency.

For each method we sweep step size, pulse count, and window. We report the resulting fronts and their exact hypervolume, the volume of objective space the front dominates, which is cheap to compute with three or four objectives.

Four checks decide the outcome.

The first compares summed, alternating, palindromic, and projected guidance at an equal pulse budget, with the gap broken down by gradient cosine.

The second is an order-swap test. We regenerate every prompt with the concept order reversed in the schedule and measure how much the per-object presence changes. Palindromic sequences should cut this asymmetry relative to plain alternation; if they do not, the commutator argument is wrong for this setting.

The third places a fixed small number of pulses at different points of the trajectory. If the best placement falls near the measured commitment step, and ideally near the theoretical speciation time, the timing component holds. If the best placement is spread across the trajectory, it is refuted.

The fourth tests whether the probe-gated schedule reaches Attend-and-Excite's caption similarity with substantially fewer backpropagations, which would close part of its reported latency gap.

A positive result is a front that dominates both weighted sums and projection on conflicting prompts, together with a measurable reduction in order bias. A null result, in which projection matches us, would still be informative: it would show that the decoupling analogy adds nothing beyond standard multi-objective tools for this problem.