Many training-free style transfer methods for image diffusion models (generators that turn random noise into an image by denoising it step by step) let the image being generated "look at" a style reference image inside the model's attention layers. Attention is the operation in which each location in the image gathers information from other locations, weighted by how similar their features are. This borrowing is what transfers the reference's colors and brushstrokes. It also causes content leakage: the reference's objects, silhouettes, and background reappear in the output, because any patch of the generated image can match and copy any individual patch of the reference. Current fixes add work on top of dense attention. One adds an extra guidance trajectory, another runs a binary search that takes about 1.5 minutes per image. Dense attention already costs time and memory that grow with the square of the number of image locations.
We propose to change what the reference exposes, not to suppress leakage after the fact. At the coarse layers of the network, where leakage starts, the generated image should not see the reference's individual patches at all. It should see only their averages along each row and each column. These are the reference's "axial marginals": $H+W$ summary tokens in place of $H\times W$ patch tokens. A row average mixes the object with everything else in that row, so it cannot reproduce a cat's head. It still carries the palette and the dominant textures. The same change shrinks the attention cost of reading the reference from $O(N^2)$ to $O(N(H+W))$, where $N=HW$. Leakage control then lowers inference cost.
Core Claim
We claim that content leakage in attention-sharing style transfer is governed by spatial grain. Leakage depends on how small a piece of the reference the generated image can address at each network resolution. We further claim that limiting coarse resolutions to axial marginals, while keeping dense patch-level access only at the finest resolutions, matches the leakage suppression of iterative methods like Only-Style. We expect it to do so in a single pass, at lower attention cost than plain shared attention, and with style fidelity no worse than StyleAligned.
The claim is falsifiable in two directions. If row and column averages still leak, for example by transferring the reference's horizon line or vertical layout as "streaks", the axial channel has its own failure mode, and we will measure it. If simpler grain reductions match it, such as one global average or scrambling the reference image, the axial structure adds nothing.
Relation to Prior Work
The proposal starts from the shared-attention recipe of Style Aligned. That method lets generated images attend to a reference's keys and values, and it normalizes queries and keys with AdaIN to reduce "positional collisions", where reference elements land on matching coordinates. We keep its normalization and change only the set of reference tokens.
Three leakage methods motivate the work by showing the problem is real and costly. StyleKeeper swaps queries to build a negative guidance direction away from reference content. It also reports that sharing attention in down-blocks and the bottleneck causes severe leakage, which is direct evidence that coarse layers are the source. Only-Style locates the reference subject, then searches for a suppression strength by checking patch similarity at the midpoint of denoising. It achieves clean results at roughly 1.5 minutes per image. Z* reweights cross-attention with a scalar and names memory and regional control as open problems. All three keep dense patch-level access to the reference and add computation. We remove patch-level access where it does harm.
Layer-routing work constrains the design by giving conflicting accounts of where content and style live. DEADiff and SCAdapter send content to coarse layers and style to fine ones. B-LoRA finds content and style separated even within the up-blocks of SDXL, at Block 4 versus Block 5. Our per-resolution grain schedule tests these accounts directly. The schedule is a variable we sweep, not a fixed assumption.
The method borrows from efficient attention. CSWin Transformer and the axial self-attention in UniST factorize attention along rows and columns for speed. UniST does this inside a style transfer model but never measures leakage. We apply the factorization to the reference's key set rather than to each query's neighborhood, and we use it for disentanglement as well as speed. From Log-linear Sparse Attention we take the practice of reweighting pooled tokens by the area they summarize, so that coarse summaries are not under-attended.
Research Program
At each shared self-attention layer of SDXL at resolution $H\times W$, we keep the target image's own keys dense. We replace the reference keys $k^{ref}_{x,y}$ (and likewise the values) with row and column averages. Here $x$ indexes the $W$ columns and $y$ indexes the $H$ rows:
$$\bar k^{row}_y=\frac{1}{W}\sum_{x} k^{ref}_{x,y},\qquad \bar k^{col}_x=\frac{1}{H}\sum_{y} k^{ref}_{x,y}.$$
A target query $q$ then takes one softmax over its own $N$ keys together with these $H+W$ marginal tokens. The logit for a row token is $q^\top \bar k^{row}_y/\sqrt{d}+\log W$, where $d$ is the key dimension. The $\log W$ term gives a row token the attention mass that its $W$ underlying patches would jointly receive if their keys were identical (and $\log H$ does the same for column tokens). This keeps the overall share of style borrowing comparable to dense sharing, so that we do not suppress leakage simply by suppressing style.
We generalize the construction into a family of channels indexed by grain. Dense patches are the finest grain. Strips of width $s$, in the manner of CSWin stripes, come next, then axial marginals. A single global average, essentially AdaIN statistics, is the coarsest. For each SDXL resolution level we choose one grain, and the schedule across levels is the object of study.
The program has three parts. The first is a mapping study. Over the full grain-by-resolution grid we measure leakage and style fidelity, which shows where along the hierarchy patch-level access becomes harmful. This also adjudicates between the StyleKeeper, B-LoRA, and DEADiff accounts. The second is a failure-mode study of axial channels. We build references with strong one-dimensional layout, such as horizons, columns, and portraits centered in a vertical strip. We then test whether row or column marginals imprint that layout on the output, which addresses the open question of whether axial factorization reduces or creates positional collisions. If streaks appear, we test random-offset strips, meaning strips whose boundaries shift each denoising step, as the fix. The third is a cost study. We generate at 1024 and 2048 pixels and report attention FLOPs, peak memory, and wall-clock time under FlashAttention-style kernels. Gather and reshape overheads can erase theoretical savings, so measured time is what we report.
Baselines
The naive reference point is plain StyleAligned shared attention at all layers with its AdaIN normalization (Style Aligned). It is exactly our pipeline with the dense-patch channel everywhere, so any difference isolates the channel. On the Only-Style evaluation set it reports a content leakage score of 0.231, where the no-leakage level is 0.21, and a 47% LLaVA no-leakage rate, as measured in Only-Style. Its DINO style similarity is 0.829 as reported in B-LoRA. We also include a no-reference run that uses the text style prompt only, which fixes the floor for both leakage and style.
The strongest existing leakage method is Only-Style. It reports a content leakage score of 0.215 and a 68.3% LLaVA no-leakage rate on its 400-pair set, at about 1.5 minutes per image on an RTX 3090. We beat it if, on the same set and SDXL checkpoint, we reach a content leakage score at or below 0.215 and a LLaVA rate at or above 68.3% in a single pass. We also require DINO style similarity no lower than StyleAligned's. A tie on leakage with an order-of-magnitude speedup still counts as a meaningful win.
The strongest structural competitor is StyleKeeper. We run it in two forms. The first is its layer rule alone: dense sharing confined to up-blocks. This is the closest "simpler" version of our idea. The second is the full method with negative query guidance. No reported result was found on Lacuna for StyleKeeper on the Only-Style set, so we rerun it.
We include B-LoRA as the style-fidelity ceiling. It reports DINO similarity of 0.881, but it requires per-image tuning, so we report its tuning time separately.
All training-free methods run on the same SDXL checkpoint, seeds, sampler, step count, and prompts. Each method receives the same hyperparameter search budget of 20 configurations, selected on a held-out set of 40 pairs, and all are timed on the same GPU.
Our key ablations isolate the axial structure. We swap marginals for global averaging at the same layers, for strips of varying width, and for marginals at every layer instead of coarse layers only. A further variant removes the $\log W$ area correction.
Why not just…?
Why not just restrict dense sharing to the up-blocks, as StyleKeeper does? This is the most serious threat, because it costs nothing to implement. However, StyleKeeper still needed negative query guidance on top of the layer rule, which suggests the rule alone leaves residual leakage. B-LoRA's finding that Block 4 within the up-blocks carries content points the same way. The "up-blocks only" baseline measures this directly. If it matches our leakage and style scores, we will report that the grain schedule adds only compute savings.
Why not just share global style statistics (AdaIN) at coarse layers? Global averaging is the coarsest grain in our family and cannot leak layout at all. Its risk is the reverse failure: it flattens style to one mean vector per layer. That would be consistent with the color dominance B-LoRA reports, where a strong palette overrides content identity. The global-average ablation, run at the same layers, decides whether the extra $H+W$ tokens retain measurable style fidelity, measured by DINO similarity and a user study. If they do not, axial marginals are unnecessary.
Why not just scramble or blur the reference image before encoding it? Degrading the reference destroys objects at scales above the patch size, as in the curriculum of D-VST. Scrambling has two weaknesses. It uses one spatial scale for the whole network, and patch seams can be read as part of the style. Both are empirical claims. We include patch-scrambled references at three patch sizes as a baseline and inspect grid artifacts in the outputs.
Why not just use FlashAttention or token pruning for the cost problem? FlashAttention reduces memory but not FLOPs. Pruning, as in FastV, drops tokens that may carry fine strokes. Our cost claim is secondary in any case: the main contribution is that leakage control need not add cost. We report all timings with fused kernels so the speedup is not overstated.
Proposed Experiments
The primary evaluation uses the Only-Style 400-pair set, scored with its content leakage metric and LLaVA critique protocol. We add the StyleAligned 100-prompt set for style consistency, and DINO cosine similarity for style fidelity. For image-guided transfer we use 20 MS-COCO content images crossed with 40 WikiArt styles, following SCAdapter, scored with LPIPS and CLIP alignment.
For the axial failure mode we construct a test set of about 100 references with dominant one-dimensional structure. On it we define a layout leakage score. It is the correlation between row and column histograms of segmentation masks from the reference and from the output. We run it with an off-the-shelf segmenter and report it alongside random-pair controls. A user study of 800 pairwise judgments compares us against Only-Style and StyleAligned, matching the Only-Style protocol.
Three outcomes decide the project. First, at the best grain schedule, does the method sit at or below Only-Style's 0.215 leakage while staying within noise of StyleAligned's style similarity? Second, does it beat both the up-blocks-only rule and global averaging on the joint leakage–style frontier? If it beats neither, the axial idea is rejected. Third, at 2048 pixels, does it reduce measured attention time and peak memory relative to dense shared attention? The grain-by-resolution map is a contribution whatever the outcome, because it shows where in SDXL patch-level reference access starts copying objects.
References
- Style Aligned Image Generation via Shared Attention
- StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance
- Only-Style: Stylistic Consistency in Image Generation without Content Leakage
- Z*: Zero-shot Style Transfer via Attention Reweighting
- DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations
- SCAdapter: Content-Style Disentanglement for Diffusion Style Transfer
- Implicit Style-Content Separation Using B-LoRA
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
- Two Birds, One Stone: A Unified Framework for Joint Learning of Image and Video Style Transfers
- Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers
- D-VST: Diffusion Transformer for Pathology-Correct Tone-Controllable Cross-Dye Virtual Staining of Whole Slide Images
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models