Many current face-swap systems no longer paste a new face onto a head. They erase the face region and have a generator redraw it, conditioned on the new person's identity, so that the new face blends into the real hair, neck and lighting around it. The leading family of face-forgery detectors works by finding the thin seam where a pasted face meets the real image. When the generator draws the seam itself, that seam stops being a reliable clue. Our claim is that the evidence does not disappear. It moves. The clue stops being a few pixels wide at the edge and becomes a whole-region disagreement: the redrawn face carries the statistics of one source, the generator, while the real neck, ears and background carry the statistics of another, the camera. Detecting that disagreement means comparing pixels that lie hundreds of pixels apart, so it is a long-range problem, not a local one.
We propose a detector built around this shift. It computes affinities, meaning a similarity score between every pair of distant image locations, on a high-frequency noise-residual signal rather than on color and semantics. Each pair is trained to say whether the two locations came from the same source. The supervision comes from cheap "self-inpainted" pseudo-fakes: real faces whose face region has been redrawn by a single off-the-shelf diffusion inpainter. We then test the detector on held-out identity-guided inpainters it never saw during training.
Core Claim
Identity-guided inpainting, as done in DiffSwap and FaceInpainter, turns face swapping into conditional hole-filling. Both papers show this removes visible boundary artifacts. DiffSwap reports 98.54% identity retrieval and an FID of 2.16 on FaceForensics++. These systems are the threat model, and they motivate the work. They supply no detection method.
On the detection side, the boundary detectors share one assumption: the forgery evidence sits in a thin band where the blending mask crosses 0.5. Face X-Ray builds this into its training target $B = 4M(1-M)$, where $M$ is the blending mask. FakeFormer adds attention to "vulnerable" boundary patches. Forensics Adapter steers a frozen CLIP encoder toward the seam.
FakeFormer itself documents a "quality paradox": detection accuracy drops as fakes become smoother at the boundary. COCO-Inpaint adds two findings. Vision transformers beat convolutional networks on diffusion inpainting. Pixel-level localization also collapses to about 31.9% F1 when the detector faces an unseen inpainter. Together these suggest that the useful signal for inpainting is global and statistical rather than a local edge.
Our claim follows. For inpainting-based face forgeries, a detector that explicitly compares low-level source statistics between the generated interior and distant authentic context will generalize across generators better than a boundary detector. The gain should come from the long-range comparison itself. Training on inpainted pseudo-fakes alone should not account for it.
Research Program
Detector design. The detector has two streams over a shared backbone. One is a standard RGB stream; we use a ViT, the architecture COCO-Inpaint found strongest. The other is a residual stream, made by subtracting a denoised copy of the image and passing the remainder through a few convolutions. The convolutions preserve the high-frequency detail that FakeFormer shows plain transformers tend to smooth away.
On the residual features we compute a pairwise affinity map $A_{ij} = \sigma(\phi(r_i)^\top \phi(r_j))$. Here $r_i$ is the residual feature at location $i$, $\phi$ is a small learned projection, and $\sigma$ is the logistic function. We train $A_{ij}$ against $s_{ij}$, which equals 1 when locations $i$ and $j$ come from the same source and 0 otherwise. The loss is binary cross-entropy summed over all pairs outside a narrow band around the mask edge.
Excluding that band settles a disagreement in the literature. From Specificity to Generality argues that pixels on the boundary are mixed and noisy and drops them from its contrastive loss, while Face X-Ray treats the boundary as the entire target. Our target is about whole regions, so boundary pixels can be ignored without losing supervision.
Keeping full-pair comparison affordable. Comparing every pair of locations at a fine stride costs quadratically, which is too much at forensic resolutions. We therefore reuse the recurrent criss-cross attention of CCNet. Each location attends only to its own row and column. Two passes connect any pair of locations, and CCNet reports this uses 11x less memory than full non-local attention.
The image-level score and the predicted region mask are read off a spectral partition of $A$, a split of the affinity map into its two most self-similar groups of locations. The predicted region is whichever group disagrees with the larger, background-anchored group.
Training data. We build self-inpainted pseudo-fakes from real FaceForensics++ frames. We mask the face region with masks of random shape and dilation, then redraw it with a single generic inpainter (SD1.5 inpainting, one of the COCO-Inpaint engines). We use no identity conditioning, or the frame's own identity. This choice matters. Because the person does not change, the detector cannot take a shortcut through identity mismatch. It has to learn the source disagreement.
Secondary probe. A second probe tests whether identity cues outside the mask add anything. It compares face-recognition embeddings of the inner face with an embedding predicted from the periphery: ears, hairline and jaw outline. We report it separately because it can only help on identity-replacing forgeries. ForgeryNet shows that identity-replaced and identity-preserving forgeries transfer poorly to each other: about 64% accuracy when training on one and testing on the other.
Baselines
Naive reference. The do-nothing reference is the same ViT trained as a plain real/fake classifier on standard FaceForensics++ fakes. It shows how much of the problem a generic classifier already handles.
Boundary detectors. These are the main conceptual competitors. We run Face X-Ray on its own blended-image training data, and FakeFormer trained on self-blended pseudo-fakes, as its authors did. FakeFormer is the strongest boundary-centric model, reporting 94.45% AUC on Celeb-DF v2 with 23M parameters. Because our model would be trained on different pseudo-fakes, we also retrain both on our self-inpainted pseudo-fakes. This separates the effect of the data from the effect of the architecture.
Strongest general method. This is From Specificity to Generality. It reports 87.8% average AUC on DF40 and 85.2% average AUC on DiFF, the diffusion facial forgery set, which is the closest existing test of diffusion-made faces.
Foundation-model baseline. Forensics Adapter represents the "large global model already handles it" position. Its file entry gives relative gains (about 7% average AUC; +10.7% on DFDC) but no reported absolute AUC on Lacuna. We will rerun it rather than quote a number.
Inpainting-as-probe baseline. EARG-Net feeds the residual of a pretrained inpainter into a dual-stream transformer. It is the closest existing use of inpainting in forensics, but it reports no specific numbers on Lacuna.
Fair-comparison protocol. Every model gets the same backbone size class, the same training identities, the same total GPU-hour budget, and the same number of hyperparameter trials, tuned on a validation split drawn only from the training generator.
Success criterion. Beating the field means two things. First, exceeding 85.2% average AUC on DiFF. Second, beating every rerun baseline, including the boundary detectors retrained on our pseudo-fakes, on the held-out identity-guided inpainting test set described below. Matching baselines on classic FaceForensics++ is enough there.
Ablations of the key ingredient. Four ablations isolate it. (1) Supervise the affinity on RGB features instead of residuals. (2) Restrict attention to a local window of fixed radius. (3) Replace the pairwise source target with Face X-Ray's boundary target, keeping everything else equal. (4) Remove the edge-band exclusion.
Why not just…?
Why not just retrain Face X-Ray or FakeFormer on inpainted pseudo-fakes? This is the most serious threat. If self-inpainted data is the whole story, a boundary detector retrained on it should catch up. The retrained baselines above are designed to test exactly this. Our hypothesis predicts the retrained boundary detectors improve on inpainters similar to the training one but fall behind on held-out generators whose redrawn region extends well past the face outline. In those cases no consistent seam sits where the boundary target says it should. If they match us across held-out generators, the architectural claim fails, and the honest contribution shrinks to the data recipe.
Why not just use a global foundation model such as CLIP with an adapter? Global attention is already present there. However, the file notes that CLIP entangles semantics with low-level statistics, which is why Forensics Adapter needs boundary-targeted losses at all. Our RGB-versus-residual ablation and the rerun Forensics Adapter test whether attention over semantic features can see source disagreement. Being global is not the same as comparing the right signal.
Why not just use EARG-Net-style reconstruction residuals? The reconstruction probe assumes the probing inpainter cannot reproduce the tampered region naturally. When the forgery was itself made by a diffusion inpainter, the probe may redraw the fake as smoothly as it redraws real content. EARG-Net also reports that its predicted masks over-extend. We test this directly on the held-out inpainting set.
Proposed Experiments
Training and test sets. We train once on FaceForensics++ real frames plus self-inpainted pseudo-fakes. Test sets are DiFF, DF40, Celeb-DF v2 and DFDC. We add a held-out identity-guided inpainting set, made by running DiffSwap and the other COCO-Inpaint engines (SDXL, Flux.1-Fill, BrushNet, PowerPaint) on FaceForensics++ identities. All of these generators are unseen during training. For localization we report pixel-level F1 against the known inpainting masks, so results can sit next to COCO-Inpaint's roughly 31.9% cross-generator pixel F1. We also run the detector on COCO-Inpaint itself to check whether the source-comparison idea carries beyond faces.
First decisive check: seam-strength sweep. We generate one set of fakes along a continuum. At one end the face is pasted and feathered. At the other it is fully inpainted, with the mask dilated progressively into hair and neck. Boundary detectors should degrade along this axis, reproducing the quality paradox. Our detector should stay roughly flat. A flat curve for the retrained boundary baselines would refute us.
Second decisive check: attention range. We cap the distance over which affinities are computed and plot AUC against the cap, separately for blended and inpainted fakes. Our claim predicts that inpainted fakes need long ranges while blended fakes saturate at short ones. That crossover is the direct evidence that the long-range dependency is doing the work.
Third check: compression. We repeat the evaluation under JPEG recompression matching FaceForensics++'s c23 and c40 settings. Residual statistics are the known weakness of boundary methods under compression, and we report where our method breaks rather than claiming robustness in advance.
References
- DiffSwap: High-Fidelity and Controllable Face Swapping via 3D-Aware Masked Diffusion
- FaceInpainter: High Fidelity Face Adaptation to Heterogeneous Domains
- Face X-Ray for More General Face Forgery Detection
- FakeFormer: Efficient Vulnerability-Driven Transformers for Generalisable Deepfake Detection
- Forensics Adapter: Adapting CLIP for Generalizable Face Forgery Detection
- COCO-Inpaint: A Benchmark for Image Inpainting Detection and Manipulation Localization
- From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes
- CCNet: Criss-Cross Attention for Semantic Segmentation
- ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
- EARG-Net: Edge-Aware Reconstruction-Guided Network for Image Manipulation Detection and Localization