TL;DR
RefGAP is a training-free, inference-time correction that fixes a major bottleneck in reference-guided diffusion editors: the "reference-attention gap." By dynamically adjusting attention logits based on real-time measurements during the forward pass, it significantly boosts identity fidelity across various models and tasks without requiring per-model manual tuning.
What Problem Does This Paper Address?
Modern diffusion-based editors (like those used for face swapping or virtual try-on) often use an "in-context" design, where reference image tokens and target image tokens share the same attention sequence. You would expect the model to easily "see" the reference, but in practice, it often ignores it.
The authors discovered that in models like LoomVideo, the regions being edited sometimes assign less than 1% of their attention mass to the reference image. This lack of attention leads to poor reproduction of the reference's identity. While researchers have tried to fix this by adding fixed "biases" to the attention mechanism, these usually require tedious, manual tuning for every new model or task. Furthermore, simply boosting reference attention everywhere can cause "leakage," where the reference's features bleed into parts of the image that were supposed to remain unchanged.
Methods
RefGAP introduces a measurement-driven, two-sided correction that operates entirely during inference.
- Measurement: During the forward pass, the system partitions attention keys into "reference" tokens and "rest" tokens. It calculates $\rho$, the fraction of attention mass a query assigns to the reference.
- Two-Sided Correction: Using the edit mask, RefGAP identifies "edit" regions (to be changed) and "keep" regions (to be preserved).
- Dynamic Logit Offsets: Instead of a fixed boost, RefGAP applies a logit offset that adapts to the measured attention mass.
- Edit Regions: It adds a positive offset to strengthen reference usage. If the model is ignoring the reference (low $\rho$), the boost is stronger.
- Keep Regions: It adds a negative offset to suppress reference influence, ensuring the background or original context stays intact.
- Selective Layer Intervention: The authors found that models typically have a "reference-reading band," often in the second half of the network. They developed a rule based on two statistics—selectivity ratio ($B$) and log-inverse edit mass ($\bar{b}_{edit}$)—to automatically determine which layers need correction.
Datasets
The authors evaluated RefGAP across seven different diffusion editors using several benchmarks:
- Head Swapping: HeadSwapBench (1,040 test clips).
- Face Swapping: FaceSwap subset of FaceForensics++ (1,000 target-source pairs).
- Virtual Try-on: 90 clips from the ViViD dataset and 100 images from VITON-HD.
- Background Replacement: 150 ground-truth videos from HeadSwapBench paired with scene references from the SUN397 dataset.
Key Insights and Ideas
- The Attention Bottleneck: The core "aha moment" is quantifying the reference-attention gap. The fact that some state-of-the-art models assign nearly zero attention to the reference explains why they struggle with identity fidelity.
- Online Signal vs. Offline Tuning: By using the model's own internal attention mass as a feedback signal, RefGAP eliminates the need for the "strength sweeps" typically required for training-free methods.
- Regional Duality: Successful editing requires a push-pull mechanism. You cannot effectively increase fidelity in the edit region without simultaneously protecting the keep region from reference leakage.
- Architecture Agnostic: Remarkably, two global coefficients ($\gamma_{edit}=0.6, \gamma_{keep}=0.8$) calibrated on a small validation set were found to generalize across vastly different architectures, including FLUX.2, VACE, and OmniGen2.
Key Results
- Identity Boost: RefGAP improved identity similarity (AdaFace) across all seven tested approaches. In head swapping, the "replacement rate" (how often the output looks more like the reference than the original subject) reached at least 86% for every model.
- Qualitative Gains: As seen in Figure 5, the "Ours" rows show much sharper adherence to the reference's facial features compared to the "Base" models.
- Efficiency: The fused implementation is highly efficient, adding only 0.5% to 7.9% overhead for most video models. While the overhead is higher for image-only FLUX models (up to 68%), the absolute time increase is less than half a second per image.
- Human Preference: In human evaluations, RefGAP was overwhelmingly preferred for reference fidelity across head swapping, face swapping, and virtual try-on.
Limitations and Future Work
- Identity-Attribute Trade-off: There is a natural tension between identity fidelity and preserving the original pose or expression. RefGAP prioritizes identity, which can lead to slight increases in keypoint errors. The authors suggest that delaying the correction to later denoising steps can mitigate this, but it's a delicate balance.
- Task-Specific Failures: While RefGAP excels at identity-related tasks, it struggled with background replacement on certain models (like Qwen-Image-Edit). This suggests that scene-level references might interact with attention mechanisms differently than object-level references.
- Profiling Requirement: While RefGAP avoids per-task tuning, it still requires one initial "profiling" run on a new model to identify the correct layer set for intervention.
Summary updated