Text-to-image diffusion models often put an attribute on the wrong object. Ask for "a blue apple and a green vase" and the vase may come out blue. The usual fix is to stop the image region belonging to one object from attending to the words for another. Doing that forces a binary decision at every image location: this patch belongs to the apple, or it does not. We propose to make that decision the way neuroimagers decide whether a brain voxel responds to a stimulus. Each image location is treated as a voxel, the model's many attention readouts are treated as repeated noisy measurements, and a regression with a significance test labels each location as confidently entity A, confidently entity B, or undecided. Only confidently assigned locations are masked. Undecided locations keep full attention, and those are often the contact zones where two entities touch or interact.
The bet is that two failures come from the same blunt decision. Leakage happens when a location the model has effectively committed to one entity still reads the other entity's attributes. Lost interactions (two penguins asked to huddle that end up standing apart) happen when ambiguous boundary locations are forced into one entity's mask. A test that tells "committed" apart from "ambiguous" should reduce leakage without severing the regions where entities meet. It should also replace the hand-tuned thresholds and multipliers of current methods with one interpretable error-rate setting.
Background and the gap
Statistical parametric mapping (SPM), the standard toolkit for functional MRI, fits a separate linear regression $Y = X\beta + \epsilon$ at every voxel. $Y$ is the measured signal, $X$ is a design matrix of hypothesized explanatory timelines plus nuisance terms such as head motion, and $\beta$ holds the fitted weights. Linear contrasts $c^T\beta$ then isolate one effect, for example "pain viewed minus neutral face", and the result is a map of t-statistics thresholded with correction for testing many voxels (Statistical Learning Methods for Neuroimaging Data Analysis with Applications). The method's value lies in three ingredients, none of them biological: an explicit noise model, nuisance regressors that soak up confounds, and principled thresholds. Calhoun et al. compared model-based GLM maps with data-driven spatial ICA on the same visual task. They found the GLM's main advantage was its formal thresholding and sensitivity, while purely data-driven maps lack a principled cutoff. That finding motivates the present proposal.
Spatial attention masking supplies the mechanism we modify. FreestyleNet sets attention scores outside a user mask to $-\infty$. Cross Attention Control injects localized prompts into boxes with a scale $\lambda$. Seg2Any blocks cross-entity self-attention only in FLUX middle layers 20–38. SynGen optimizes overlap losses on attention maps, at about 9.7 seconds per image against 4.0 seconds for plain Stable Diffusion.
The closest work is DeLeaker. It needs no user masks: it estimates entity masks from early cross-attention, zeroes strong cross-entity attention, and boosts self-attention within each entity. Its own limitations define our target. It resolves only 46.07% of leakage cases on its benchmark. It sometimes erases requested interactions. Its masks come from heuristic cutoffs with no estimate of how reliable each location's assignment is.
Our method uses the same family of signals as DeLeaker but differs in two ways. First, each location's assignment is computed as a contrast with an error bar estimated from disagreement across attention heads, layers and timesteps. Second, masking is three-way, so ambiguous locations are deliberately left unmasked instead of being forced into a binary partition.
Core claim
For each spatial latent token $i$ and each entity $e$ in the prompt, we collect the cross-attention mass from token $i$ to $e$'s words across $S$ readouts: attention heads × a band of layers × the first few denoising steps. These $S$ values form the "time series" $y_i \in \mathbb{R}^{S}$, by analogy with a voxel's signal. We fit
$$y_i = X\beta_i + \epsilon_i,$$
where $X$ contains one indicator column per entity and nuisance columns. The nuisance columns are per-readout global attention mass, attention-sink and padding-token mass, and a layer-depth trend. They play the role motion regressors play in fMRI: they keep readouts that are simply louder from masquerading as entity evidence. For an entity pair $(A,B)$ the contrast $c = e_A - e_B$ gives a statistic
$$t_i = \frac{c^T\hat\beta_i}{\hat\sigma_i\sqrt{c^T(X^TX)^{-1}c}},$$
where $\hat\sigma_i$ is the residual standard deviation at token $i$. Readouts are correlated across heads, so we do not trust the parametric null. Instead we calibrate $t_i$ by sign-flipping permutations over readout blocks, then control the false discovery rate across the latent grid at a fixed level $q$.
Tokens significantly favoring A receive A's mask, and tokens significantly favoring B receive B's. All remaining tokens, the undecided set, attend globally. Masks are applied to cross-attention and to cross-entity self-attention. The fit is closed-form least squares on a few thousand small regressions, so it adds well under a second, in contrast to SynGen's optimization loop.
The claim is falsifiable on three counts. At equal image quality, the three-way statistical masks should resolve more SLIM leakage cases than DeLeaker. They should preserve requested interactions more often. And a single $q$ should transfer across prompts and across the two backbones without retuning, whereas $\lambda$ or $\alpha$ must be retuned.
Research program
The first stage is diagnostic. On FLUX.1-dev we run SPM-style maps over unmodified generations from SLIM prompts and test whether leakage, as human annotators mark it, is concentrated in tokens with large $|t|$ toward the wrong entity. If annotated leakage instead sits in undecided tokens, the core hypothesis is wrong and the masking stage is not worth building. We report this outcome either way.
The second stage builds the intervention and studies its free choices: which readouts enter $y_i$ (layer band, number of steps), which nuisance columns to include, and $q$. Seg2Any's 20–38 layer finding and DeLeaker's early-step localization give the starting point. Following Wilson and Niv, who showed that highly correlated regressors make maps robust to parameter choice but poor at separating model variants, we measure how stable the masks are across these choices. We present that stability as a property to quantify, not one to assume.
The third stage ports the method to Stable Diffusion. There it can be compared with SynGen and Cross Attention Control on their own benchmarks.
Baselines
The naive reference point is unmodified FLUX.1-dev, the model used to build SLIM, sampled with identical seeds. It fixes the leakage rate any method must reduce and the LPIPS distance of zero against which structural change is measured.
The strongest existing competitor is DeLeaker. It addresses the same problem, mask-free leakage removal on FLUX, on the same benchmark, and reports 46.07% mitigation, LPIPS 0.22, and human preference in 67.8% of 980 responses. The prior baselines it reports are 3DIS at 29.08% and RAG-Diffusion at 17.55%. We will rerun DeLeaker on the 1,130 SLIM samples with our seeds and hardware, and tune its $\alpha$ and threshold on a held-out prompt split with the same number of configurations we allow for $q$ and the layer band. Beating it means higher mitigation at LPIPS no worse than 0.22 and a human preference majority in a side-by-side study.
On Stable Diffusion, SynGen is the strongest binding method, with a 94.76% proper-binding score on the Attend-and-Excite benchmark. We expect to approach that score, not exceed it. The comparison is about matching most of the gain at roughly baseline latency.
Cross Attention Control reports 67.27% against 49.91% for vanilla SD on CC-500. It uses given boxes and so receives privileged layout information. We include it as an upper reference, not as a like-for-like competitor.
Two ablations isolate the key ingredient. The first replaces $t_i$ with the raw mean attention margin (entity A mean minus entity B mean) and a tuned threshold. It keeps the three-way scheme but removes the noise estimate and nuisance regressors. The second keeps the statistical masks but masks the undecided set as well, which isolates the contribution of leaving ambiguous zones open.
Why not just…?
Why not just dilate or soften binary masks to leave room at boundaries? A fixed-width band around DeLeaker-style masks also leaves contact zones unmasked, and this is a real threat. Our claim is that the undecided set is not a uniform band. It should widen where entities physically interact and shrink where they are cleanly separated. We will run a morphological band matched in area to our undecided set on every image. If it matches our mitigation and interaction scores, the statistical machinery is unnecessary and we will say so.
Why not just threshold the attention margin, like an Otsu cut on averaged maps? This is the first ablation. A margin ignores that some locations receive consistent attention across heads while others receive large but erratic attention. Calhoun et al.'s comparison suggests that cutoffs without a noise model do worse at separating weak true signal from diffuse background, but that evidence comes from fMRI, not diffusion models. If the tuned margin equals the t-map at matched tuning budget, the remaining contribution reduces to one tuning-free setting $q$. We will test whether that alone transfers across backbones.
Why not just optimize attention losses as SynGen does? It works: 94.76% binding on Attend-and-Excite. But it costs about 144% extra latency and depends on a correct syntactic parse. Our method is closed-form and parse-light, since it needs only the entity spans. The Stable Diffusion comparison will report binding against wall-clock time so the trade-off is explicit.
Decisive experiments
The main experiment is SLIM on FLUX.1-dev. We report mitigation rate, LPIPS, and a human preference study run as in DeLeaker against all baselines and ablations.
Interaction preservation has no benchmark in the file, so we will construct one. It will contain SLIM-style entity pairs combined with explicit relational verbs ("huddling", "holding", "leaning on"), and human raters will judge both leakage and whether the relation is depicted. This is where DeLeaker's acknowledged failure should show up, and where the second ablation, which also masks the undecided set, should hurt.
For transfer, we fix $q$ from the FLUX tuning split and apply it unchanged on Stable Diffusion with the Attend-and-Excite, CC-500 and DVMP prompts. We compare against DeLeaker's parameters, also carried over unchanged.
The diagnostic from the first stage serves as a precondition. If leakage annotations do not align with confidently misassigned tokens, the masking results are uninterpretable and the proposal fails at the first step.
References
- Statistical Learning Methods for Neuroimaging Data Analysis with Applications
- fMRI Activation in a Visual-Perception Task: Network of Areas Detected Using the General Linear Model and Independent Components Analysis
- Freestyle Layout-to-Image Synthesis
- Localized Text-to-Image Generation For Free via Cross Attention Control
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment
- DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models
- Is Model Fitting Necessary for Model-Based fMRI?