Donor-Centered Windows: Refining Coarse Attention Transfer for High-Resolution Inpainting

Fast high-resolution inpainting systems share one trick. They fill the hole at low resolution, then decide which visible region each hole location should copy from, and they make that decision only once, on a tiny feature map. The same "copy from there" map is then stretched up to full resolution and used to paste sharp detail. The trick is cheap, but the stretched map is blocky. At 4K, one low-resolution decision governs a block of tens of pixels on each side. The fine scales cannot correct it, so pasted texture is misaligned, seams appear, and slightly rotated or rescaled patterns are copied wrong.

We propose letting each fine scale correct the coarse decision at nearly the cost of local attention. The coarse scale proposes a few donor locations for each hole region. At the next finer scale, each query attends only inside small windows centered on those donors and on its neighbors' donors, never on itself. Ordinary local windows attend around the query, so their reach is short. Placing the windows at globally retrieved donors keeps global reach while computing only local attention. Cost grows linearly with pixel count instead of quadratically, and the windows are contiguous memory blocks that GPUs and mobile chips read efficiently.

Core Claim

Coarse-to-fine attention transfer works better when it is treated as a correspondence that each scale refines, rather than a map copied once. The claim has three parts. First, refining correspondences inside donor-centered windows recovers most of the quality of dense fine-scale attention. Second, its cost is about $N k w^2$, where $N$ is the number of fine-scale hole tokens, $k$ the number of donor proposals per token (about 4 to 8), and $w$ the window side (about 5 to 9). Dense attention costs $N M$, where $M$ is the number of visible tokens, and that is infeasible at 4K. Third, the benefit comes specifically from where the windows sit, not from adding local attention. An equally sized window centered on the query should not give the same gain.

Where This Sits in Prior Work

The motivating line is cross-scale attention transfer for inpainting, the task of filling a masked region of an image. PEN-Net computes region affinity at the deepest, lowest-resolution encoder layer and passes that affinity map to shallower layers. Its authors explicitly leave scaling to higher resolution as future work. CRA computes attention once on a low-resolution feature map and reuses the scores to borrow high-frequency residuals (the sharp image minus its blurred version) from visible patches at full resolution. The file notes its transfer is "driven by low-resolution patch matching" and cannot align geometric deformations. RETHINED does the same with a NeuralPatchMatch module and attention upscaling, which reaches 8K on mobile hardware. All three fix the correspondence at one coarse scale. We keep their pipelines and replace only the step that applies the attention map at full resolution.

A second line expands the receptive field cheaply, meaning the range of the image a unit can see. CCNet restricts each pixel to its row and column and gets global reach at $O(N\sqrt{N})$. It operates at a single scale for segmentation. ATD groups tokens by content category so attention is global but linear in cost. It targets super-resolution and denoising, not filling holes from donors. Deformable attention (DAS) learns sampling offsets outside fixed kernels.

Two results constrain the design. The profiling work by Xu et al. shows that in multi-scale deformable attention, scattered sampling dominates runtime despite modest FLOPs. This is why we gather whole windows rather than scattered points. ATD reports that larger retrieval pools dilute attention, which argues for small, well-chosen candidate sets over large ones. Contextual attention supplies a reused ingredient: propagating attention between neighboring queries to keep copied patches contiguous. We use it by adding neighbors' donors to each query's candidate set.

The file's suggested Gap 1, criss-cross attention applied across scales, is the obvious combination. We think it targets the wrong cost. At the coarse scale, dense affinity is already cheap, since CRA computes it on a small map. The binding problem is that the fine scales cannot refine the coarse answer. Criss-cross attention at fine scale gives rows and columns that are mostly irrelevant to where the donor actually is.

Research Program

The operator works scale by scale. At the coarsest feature level we compute dense attention between hole tokens and visible tokens, exactly as in CRA, and keep the top-$k$ donor positions per hole token. Moving to the next level doubles the resolution. Each fine query inherits its parent's $k$ donors, scaled to the new grid, plus the donors of its four neighbors. It then runs softmax attention over the union of $w \times w$ windows centered at those positions:

$$ \text{out}(q) = \sum_{p \in \bigcup_{j \in D(q)} W_w(j)} \operatorname{softmax}_p\!\left(\frac{\langle Q_q, K_p\rangle}{\sqrt{d}}\right) V_p $$

Here $D(q)$ is the inherited and propagated donor set, $W_w(j)$ is the $w \times w$ window around $j$, and $Q, K, V$ are the query, key, and value projections. The window $W_w(j)$ is restricted to visible pixels. The new top-$k$ positions inside the windows become the donors for the next level.

At the last levels, the values are CRA-style high-frequency residuals rather than learned features. The copied detail therefore comes from real visible pixels, and the operator stays usable at resolutions it never trained on. This follows the reasoning that lets RETHINED and CRA reach 8K. Windows are fetched as contiguous tiles in a fused kernel, written in Triton for GPU and as tiled gathers for Apple Metal. We will report memory traffic as well as FLOPs, because Xu et al. show that FLOPs mislead.

Three questions are analyzed alongside quality. The first is how much the correspondence moves from coarse to fine. We measure this as the displacement between the inherited donor and the final best-matching position. The second is whether that movement predicts where one-shot transfer fails. The third is how quality trades against $k$ and $w$. This approach does not attempt to create content when no donor exists, which is CRA's acknowledged context-availability failure. We expect no gain there and will measure that directly rather than claim it.

Baselines

The naive reference point is low-resolution inpainting followed by bicubic upsampling, with no attention transfer. It uses the same coarse generator as every other system. CRA and Patch-Adapter both report that this approach blurs and leaves seams, and it sets the floor any transfer method must clear.

The strongest existing method on this question is one-shot coarse attention transfer, in the form of CRA's attention computing and transfer modules and RETHINED's attention upscaling. It is the strongest competitor because it is what currently makes 2K to 8K inpainting real-time. CRA reports about 37 ms at 2K on a GTX 1080 Ti. RETHINED reports 34.33 ms at 2048×2048 on an Apple M2 iPad Pro, against 101.4 ms for CoordFill and 3905.4 ms for MI-GAN.

For a fair comparison, we will reimplement this as a module swap. The coarse generator, training data, losses, and number of training steps stay identical, and only the transfer step changes. We will retrain both with the same hyperparameter search budget, and also report the released RETHINED and CRA checkpoints where available. On DF8K-Inpainting the file gives no reported quality numbers (no reported result found on Lacuna for RETHINED's LPIPS, L1, FID, or SSIM on DF8K). Our own reimplementation's numbers therefore serve as the bar. Beating it means lower LPIPS and high-frequency error inside the hole at 4K and 8K, at no more than 1.3 times its measured latency on the same device.

The quality ceiling is dense attention between all fine hole tokens and all visible tokens at each refinement level. It is run with memory-efficient attention on a large GPU and is feasible only up to about 2K. It tells us how much of the dense-attention gain the windows recover. A good outcome is closing at least 80% of the gap between one-shot transfer and dense attention.

The key ablation isolates where the windows sit. It uses query-centered windows with the same $w$, the same total number of attended positions ($k$ windows' worth of area), and the same kernel. If this matches donor-centered windows, the claim is false. A second ablation drops neighbor propagation, to test whether contiguity comes from it. A third substitutes learned deformable offsets for the inherited donors.

Why not just…?

Why not just compute dense attention at a finer coarse scale, with a memory-efficient kernel? This is the most serious threat. If moving CRA's attention from a 32×32 map to 64×64 or 128×128 fixes the blockiness, refinement is unnecessary. Liu et al. show memory-efficient attention and kernel fusion make dense cross-attention practical in diffusion U-Nets. However, the cost still grows with the product of hole and visible tokens, which rules out edge latency budgets. We will sweep the dense attention resolution and plot quality against measured latency on both the GPU and the M2. The proposal wins only if its curve lies above dense attention's at matched latency. If it does not, we will report that.

Why not just use shifted local windows, or criss-cross attention, at the fine scale? ATD reports that restricting attention to small local windows limits the receptive field and hurts fine textures; it beats window-based HAT by up to 0.40 dB on Urban100. Query-centered windows mostly see the hole itself, where nothing is visible to copy. CCNet's row and column paths reach the whole image, but each query sees almost none of the specific donor region. Both are run as matched-cost ablations above.

Why not just learn deformable offsets? Learned offsets can in principle find donors, but they must regress long-range displacements from features inside the hole. Xu et al. also show that scattered sampling is slow in practice. Our third ablation tests whether retrieved donors beat regressed ones at equal sample count, and compares their wall-clock time.

Why not just smooth CRA's upsampled attention map (bilinear rather than nearest)? Smoothing removes block edges but cannot move a correspondence to a better position. It will be included as a near-free baseline. Our synthetic deformation test, described below, should separate it from true refinement.

Proposed Experiments

The main benchmark is DF8K-Inpainting at 2K, 4K, and 8K with free-form masks. We report LPIPS, L1, FID, and SSIM over the whole image and inside the hole. We also report the error in the high-frequency band within the hole, since that is what transfer is meant to fix. Latency and peak memory are measured on an Apple M2 iPad Pro and on one desktop GPU. At standard resolution we report Places2 at 512 against PEN-Net's MS-SSIM of 78.09% as a sanity reference, and CelebA-HQ with 20–40% masks against the PSNR of 28.83 reported in Delving Globally.

The decisive check is a controlled deformation test with known correct answers. We take high-resolution textures from DF8K backgrounds and paste copies under known sub-patch shifts, small rotations, and scale changes of 0.8 to 1.25. We then mask one copy. Because the true donor and its warp are known, we can measure correspondence error directly. The prediction is that one-shot transfer's error grows with deformation, while donor-centered refinement stays flat up to about half a window.

A second check splits DF8K holes by how much the correspondence moved between coarse and fine levels. Our gains should concentrate in the high-movement split. Gains should vanish on holes whose content has no visible donor, which we label by retrieval similarity below a threshold. That would confirm the method fixes alignment, not content creation.

Finally, the ablation on window position, the $k$ and $w$ sweep, and the latency-matched comparison against dense attention determine whether the idea holds. Donor-centered windows must beat query-centered windows at equal cost, and must lie above dense attention on the quality-versus-latency plot.