Image generators that copy from a reference picture, such as in-context editing models that place a reference image next to the image being generated, pay a steep compute price. Attention compares every token with every other token, so its cost grows with the square of the token count, and adding a 4,096-token reference roughly doubles the sequence. ReCAST showed that most of this cost can be cut without retraining: generated tokens send over 80% of their attention to the top 25% of reference tokens, so the rest can be dropped. The same authors report a clear failure, though. Style transfer degrades, because brushstrokes, lighting and texture are spread over the whole reference, background included, rather than sitting in a few salient patches.
We propose to stop discarding the long tail of reference tokens and to summarize it instead. The reference is split into two parts. A small exact set holds the tokens that generation attends to most, as in ReCAST. The remaining tokens are grouped into a few dozen clusters, and each cluster is carried through the network as a single summary token. When the model attends to a summary token, its score gets a bonus equal to the log of the cluster's size, so that token receives roughly the total attention its members would have received together. The claim is that the missing style is exactly this dropped attention mass: the low-weight tail that is individually negligible but large in aggregate. Restoring it through mass-preserving summaries should recover style-transfer quality at nearly the token budget of hard pruning. It should also make sparse attention usable for cross-domain appearance transfer, where the reference and target share little structure.
Core Claim and Mechanism
Fix one generated token's query $q$. The reference keys are grouped into an exact set $E$ and clusters $c$, where cluster $c$ has $n_c$ members, mean key $\mu_c$ and mean value $\bar v_c$. The attention the tail receives is approximated by
$$\sum_{j \in c} e^{q \cdot k_j / \sqrt d} \approx n_c\, e^{q \cdot \mu_c / \sqrt d}, \qquad \text{output} \approx \frac{\sum_{j\in E} e^{s_j} v_j + \sum_c n_c e^{s_c} \bar v_c}{\sum_{j\in E} e^{s_j} + \sum_c n_c e^{s_c}},$$
where $s_j = q\cdot k_j/\sqrt d$ is the score for exact token $j$, $s_c = q\cdot\mu_c/\sqrt d$ is the score for summary $c$, and $d$ is the key dimension. The approximation is exact when the keys within a cluster are identical. Its error grows with how much $q\cdot k_j$ varies inside a cluster, since by Jensen's inequality the approximation underestimates the true sum, and the gap grows with that spread. In practice we add $\log n_c$ to the attention logit of summary token $c$. This is the same proportional-attention trick used for merged tokens, applied here only to the low-relevance tail.
ReCAST predicts that dropping the tail loses little mass on subject prompts. Our hypothesis is that on style prompts the dropped mass is large and diffuse, so the tail can be clustered coarsely without much error.
Cost stays close to ReCAST's. Cluster assignments are computed once, by k-means on reference keys from ReCAST's own profiling pass, at a cost linear in token count. Summary tokens are initialized as mean hidden states of their members and then processed as ordinary tokens. With $|E|$ exact tokens and $C \approx 32$ to $64$ summaries, the reference costs $|E|+C$ tokens instead of $|E|$.
Relation to Prior Work
ReCAST supplies both the motivation and the base method. We keep its profiling pass and its schedule of token budgets that change with the denoising step, and we replace its hard deletion of tail tokens with summaries. EDITBRIDGE routes high-resolution attention through correspondences taken from a low-resolution preview. It is fragile when those correspondences are wrong, and it was tested only on edits within the natural-photo domain. Our summaries need no correspondence at all, so they act as a fallback exactly where correspondence breaks down.
Trainable Log-linear Sparse Attention supplies the closest technical idea. It re-injects coarse pooled tokens so that sparse selection does not lose context, but it works within a single sequence and requires training. We apply a training-free, mass-weighted version to cross-image reference tokens. LookupViT likewise compresses global information into a few tokens. Its reported weakness on dense tasks is why we keep an exact token set for the structure-critical part instead of compressing everything.
On the cross-domain side, Cross-Image Attention transfers appearance by feeding structure-image queries against appearance-image keys and values. It pays the full product of the two token counts in cost and gives no way to prune. It also shows that when the two images share little, attention flattens, which is exactly the diffuse regime our summaries target. DiffuseIT constrains the proposal. It shows that structure lives in spatial attention keys while global style is carried by summary embeddings, which is consistent with splitting the reference into exact structural tokens and pooled style tokens.
Research Program
The work proceeds in three steps. The first is a diagnostic. On FLUX.1 Kontext, we measure the fraction of cross-image attention that falls outside ReCAST's retained set, together with attention entropy, and break both down by KontextBench task type (subject preservation, style transfer, background change). We also measure them on structure/appearance pairs from Cross-Image Attention's domains, including its mismatched pairs such as car to house. If the dropped mass on style prompts is not clearly larger than on subject prompts, the premise fails and we stop.
The second step builds the mechanism described above, with ablations over cluster count, the split between exact and summary tokens, and whether cluster assignments are refreshed. The third step ports the same mechanism into Cross-Image Attention's key/value swap inside Stable Diffusion's U-Net. This tests whether sparse exact tokens plus summaries can replace the full cross-image attention matrix at higher resolutions, where the quadratic cost actually bites.
Baselines
The naive reference points are full attention over the whole reference, which sets the quality ceiling and the latency floor, and random selection at a matched budget.
The strongest existing method is ReCAST itself, run with its released budget schedule on the same model, the same seeds and the same A100. Its reported results are 97.3% retention of DINO identity score on DreamBench++ with about 600 of 4,096 reference tokens, and latency reduced from 14.2s to 5.9s per image. There is no reported result on Lacuna for ReCAST's style-transfer numbers on KontextBench. We will therefore measure them ourselves, and that measurement is the gap we aim to close. Beating ReCAST means recovering most of the style-similarity gap to full attention on style prompts at under 15% extra latency, while matching its identity retention on DreamBench++.
For cross-domain transfer, the competitor is full Cross-Image Attention. Its reported result is a 66.4% user preference on Buildings against 20.9% for Swapping Autoencoder. Success means matching its style and structure metrics at lower memory, and scaling to resolutions where full cross-image attention runs out of memory.
We will also compare against Token Merging and self-attention-based pruning, as in ReCAST. Every method gets the same total reference-token budget. Where a method has hyperparameters, each gets the same tuning sweep on a held-out split.
The key ablation is equal-budget hard pruning. We keep $|E|+C$ exact tokens and no summaries, which asks whether a few extra exact tokens explain the gain. We also run summaries without the $\log n_c$ bonus, which isolates mass preservation from mere pooling.
Why not just…?
Why not just give ReCAST a larger budget on style prompts? This may partly work, and it is our main equal-budget ablation. If style cues really are diffuse, recovering them by adding exact tokens needs budgets that approach the full reference. Summaries should dominate on the curve of quality against token count. If they do not, the idea reduces to budget tuning and we will report that.
Why not just use Token Merging, which already applies a size-proportional attention bonus? This is the real threat, since mechanically our method is "exact top tokens plus merging on the rest." ReCAST reports that ToMe underperforms cross-image selection, because merging by similarity ignores what generation actually needs. The comparison we run is ToMe over the whole reference, against relevance-based exact tokens plus merging on the tail only. If the two tie, the contribution is the diagnostic finding rather than the mechanism.
Why not just prune and restore style with a global statistic, such as AdaIN on latents as Cross-Image Attention does? This is cheap and plausible for colour and contrast. However, brushstroke and texture layouts are spatially varying and cannot be carried by a single mean and variance. We test ReCAST plus AdaIN directly. If it matches our method on style metrics, we adopt it.
Why not just translate at low resolution and upsample? EDITBRIDGE shows that super-resolution applied without access to the source hallucinates details, and its bridge approach was preferred 65% of the time over such two-stage pipelines. We therefore include a low-resolution translation followed by upsampling only as a latency reference.
Proposed Experiments and Decisive Checks
We evaluate on KontextBench (1,026 pairs, split by task), on DreamBench++ for identity, and on cross-domain structure/appearance pairs in the style of Cross-Image Attention, graded by semantic distance in the way Interp3DData grades morph difficulty. Metrics are DINO identity score, CLIP image similarity to the style reference, VGG Gram-matrix distance for texture, structure similarity between DINO self-similarity maps of the output and the structure image, and wall-clock time and memory measured on the same hardware for every method.
Two outcomes decide the question. First, on style prompts, measured style similarity should track the recovered attention mass, and our method should close most of the gap between ReCAST and full attention. Second, on the most semantically distant pairs, the benefit should grow rather than shrink, because attention there is flatter and the tail holds more of it. A null result on either check falsifies the core claim.
References
- ReCAST: Probing Sparse Reference Use in In-Context Image Generation
- EDITBRIDGE: Towards Faithful and Efficient Ultra-High-Resolution Image Editing
- Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
- LookupViT: Compressing Visual Information to a Limited Number of Tokens
- Cross-Image Attention for Zero-Shot Appearance Transfer
- Diffusion-based Image Translation using disentangled style and content representation