Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing

Yanan Wang, Shengcai Liao, Guangyi Liu, Xiaodan LiangarXiv 2026 · arXiv

TL;DR

RefGAP is a training-free, inference-time correction that fixes a major bottleneck in reference-guided diffusion editors: the "reference-attention gap." By dynamically adjusting attention logits based on real-time measurements during the forward pass, it significantly boosts identity fidelity across various models and tasks without requiring per-model manual tuning.

What Problem Does This Paper Address?

Modern diffusion-based editors (like those used for face swapping or virtual try-on) often use an "in-context" design, where reference image tokens and target image tokens share the same attention sequence. You would expect the model to easily "see" the reference, but in practice, it often ignores it.

The authors discovered that in models like LoomVideo, the regions being edited sometimes assign less than 1% of their attention mass to the reference image. This lack of attention leads to poor reproduction of the reference's identity. While researchers have tried to fix this by adding fixed "biases" to the attention mechanism, these usually require tedious, manual tuning for every new model or task. Furthermore, simply boosting reference attention everywhere can cause "leakage," where the reference's features bleed into parts of the image that were supposed to remain unchanged.

Methods

RefGAP introduces a measurement-driven, two-sided correction that operates entirely during inference.

  1. Measurement: During the forward pass, the system partitions attention keys into "reference" tokens and "rest" tokens. It calculates $\rho$, the fraction of attention mass a query assigns to the reference.
  2. Two-Sided Correction: Using the edit mask, RefGAP identifies "edit" regions (to be changed) and "keep" regions (to be preserved).
  3. Dynamic Logit Offsets: Instead of a fixed boost, RefGAP applies a logit offset that adapts to the measured attention mass.
  4. Edit Regions: It adds a positive offset to strengthen reference usage. If the model is ignoring the reference (low $\rho$), the boost is stronger.
  5. Keep Regions: It adds a negative offset to suppress reference influence, ensuring the background or original context stays intact.
  6. Selective Layer Intervention: The authors found that models typically have a "reference-reading band," often in the second half of the network. They developed a rule based on two statistics—selectivity ratio ($B$) and log-inverse edit mass ($\bar{b}_{edit}$)—to automatically determine which layers need correction.
Figure 1: RefGAP overview. Attention-guided corrections strengthen reference attention inside the edit and suppress it outside, using two coefficients shared across approaches and tasks.

Datasets

The authors evaluated RefGAP across seven different diffusion editors using several benchmarks:

  • Head Swapping: HeadSwapBench (1,040 test clips).
  • Face Swapping: FaceSwap subset of FaceForensics++ (1,000 target-source pairs).
  • Virtual Try-on: 90 clips from the ViViD dataset and 100 images from VITON-HD.
  • Background Replacement: 150 ground-truth videos from HeadSwapBench paired with scene references from the SUN397 dataset.

Key Insights and Ideas

  • The Attention Bottleneck: The core "aha moment" is quantifying the reference-attention gap. The fact that some state-of-the-art models assign nearly zero attention to the reference explains why they struggle with identity fidelity.
  • Online Signal vs. Offline Tuning: By using the model's own internal attention mass as a feedback signal, RefGAP eliminates the need for the "strength sweeps" typically required for training-free methods.
  • Regional Duality: Successful editing requires a push-pull mechanism. You cannot effectively increase fidelity in the edit region without simultaneously protecting the keep region from reference leakage.
  • Architecture Agnostic: Remarkably, two global coefficients ($\gamma_{edit}=0.6, \gamma_{keep}=0.8$) calibrated on a small validation set were found to generalize across vastly different architectures, including FLUX.2, VACE, and OmniGen2.
Figure 2: Reference-attention profiles and layer selection. Reference-attention masses across layers on the head-swap validation split, shown on a log scale. Gray bars mark layers selected by eq. (8). The bottom-right panels report complementary statistics B and ¯bedit, with their thresholds.

Key Results

  • Identity Boost: RefGAP improved identity similarity (AdaFace) across all seven tested approaches. In head swapping, the "replacement rate" (how often the output looks more like the reference than the original subject) reached at least 86% for every model.
  • Qualitative Gains: As seen in Figure 5, the "Ours" rows show much sharper adherence to the reference's facial features compared to the "Base" models.
  • Efficiency: The fused implementation is highly efficient, adding only 0.5% to 7.9% overhead for most video models. While the overhead is higher for image-only FLUX models (up to 68%), the absolute time increase is less than half a second per image.
  • Human Preference: In human evaluations, RefGAP was overwhelmingly preferred for reference fidelity across head swapping, face swapping, and virtual try-on.
Qualitative comparison across seven models and four tasks, showing significant fidelity improvements.

Limitations and Future Work

  • Identity-Attribute Trade-off: There is a natural tension between identity fidelity and preserving the original pose or expression. RefGAP prioritizes identity, which can lead to slight increases in keypoint errors. The authors suggest that delaying the correction to later denoising steps can mitigate this, but it's a delicate balance.
  • Task-Specific Failures: While RefGAP excels at identity-related tasks, it struggled with background replacement on certain models (like Qwen-Image-Edit). This suggests that scene-level references might interact with attention mechanisms differently than object-level references.
  • Profiling Requirement: While RefGAP avoids per-task tuning, it still requires one initial "profiling" run on a new model to identify the correct layer set for intervention.

Summary updated

Abstract

Reference-guided diffusion editors struggle to faithfully reproduce user-provided references. We identify a potential bottleneck in diffusion editors: many methods provide limited reference-attention allocation. For example, in LoomVideo, edit-region queries assign less than 1% of their attention mass to the reference. We introduce RefGAP, a training-free correction that determines logit-offset magnitudes online at each layer from the reference-attention mass measured during the forward pass. Positive offsets to reference logits strengthen reference usage by edit-region queries, while negative offsets for keep-region queries limit reference-induced changes outside the edit. Two global coefficients control the correction; they are selected once on validation data from four development diffusion editors and held fixed. Across seven diffusion-based image/video editors, RefGAP improves identity fidelity in head swapping and face swapping. RefGAP achieves a fidelity-preservation trade-off comparable to separately tuned constant edit-side biases, without per-approach strength sweeps. Additional experiments on virtual try-on and background replacement evaluate transfer beyond identity editing.
BibTeX
@misc{wang2026mind,
  author = {Yanan Wang and Shengcai Liao and Guangyi Liu and Xiaodan Liang},
  title = {{Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing}},
  year = {2026},
  eprint = {2609.35708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.35708},
}
Figure