Within-Image Negatives for Weakly Supervised Forgery Detection

Face-forgery detectors that score near 99% on the dataset they were trained on often fall to near chance on a new dataset. When detectors trained on FaceForensics++ deepfakes were tested on Celeb-DF, average AUC dropped to 56.9% (Celeb-DF). A related problem appears when detectors are asked to say where an image was edited using only "real/fake" labels per image. On diffusion-inpainted faces, the best weakly supervised method reaches 37.7% IoU against 83.7% with pixel masks (Weakly-Supervised Deepfake Localization in Diffusion-Generated Images). This note argues that the two failures are linked. A detector that cannot find the edited region has no choice but to separate real from fake using whatever differs between whole real images and whole fake images. In a fixed training set, those differences include compression level, resolution, identity, and camera statistics as much as manipulation traces.

The proposal is to use weak localization as a source of better training data, not as the end goal. In a locally edited image, the unedited part is a real sample that shares the same camera, compression, lighting, and person as the edited part. If a detector can roughly find the edited region from image-level labels alone, it can contrast edited patches against untouched patches of the same image. Dataset-level shortcuts cannot help separate those two groups, so the detector is pushed toward manipulation evidence. To make the rough localization good enough to supply these within-image negatives, we borrow one tool from weakly supervised object detection (WSOD), the field that trains box detectors from image tags only. That tool is spatial likelihood voting: many overlapping windows vote on where the evidence lies, and the resulting map supervises the network.

Core Claim

Weakly supervised forgery detectors overfit across datasets partly because their positives and negatives differ in more than manipulation. Within-image negatives remove that confound for free. We predict that a patch detector trained with consensus-voted pseudo-masks and within-image contrast will do two things. First, it should narrow the weak-versus-full localization gap on the diffusion-inpainting benchmark. Second, it should improve cross-dataset AUC from FaceForensics++ to Celeb-DF, DFDC, DeeperForensics-1.0, and WildDeepfake. The causal ingredient should be the within-image origin of the negatives, not merely the presence of spatial regularization. That makes the claim falsifiable: if negatives drawn from other real images work equally well, the hypothesis is wrong.

Relation to Prior Work

Three papers motivate the gap. The diffusion-localization study above compares GradCAM, attention masks, and Patch-Forensics on an Xception backbone under image-level labels. It finds that patches beat the other two and that all three degrade across generators (worst on latent diffusion) and across datasets (CelebA-HQ to FFHQ). We start from its patch setup, but add cross-window consensus and use the localization to change the training signal; that paper only reads localization out. Towards Generalizable Deepfake Detection by Primary Region Regularization (PRLE) diagnoses the forensic version of WSOD's "part domination," where detectors lock onto the lips or nose. It masks those regions using activation maps from a zoo of pretrained detectors. PRLE regularizes a global classifier by removing evidence. We instead relabel evidence inside each image and need no model zoo. Finding Facial Forgery Artifacts With Parts-Based Detectors shows that some landmark-defined parts transfer better than the full face, but that naive pooling across parts does not help and parts alone fall short on DFDC. This constrains us: localization must be learned rather than fixed by landmarks, and the final decision must still see the whole face.

Three papers supply the method. Spatial Likelihood Voting with Self-Knowledge Distillation lets overlapping region proposals vote into a class likelihood map, then distills that map back into the backbone. It reached 57.2% mAP on VOC 2007. We reuse this vote-then-distill structure with sliding patches in place of object proposals. C-MIL smooths the multiple-instance objective early in training by averaging over large overlapping subsets and shrinking them over time. We borrow only its schedule: windows start large and shrink. The aim is to stop the voting map from collapsing onto one tiny artifact in the first epochs. Sharp Multiple Instance Learning shows that max-pooling over instances starves non-maximal instances of gradient. That is a constraint on our image-level aggregation, so we pool with a soft weighted sum rather than a max.

Two papers act as competitors with different supervision. Locality-aware AutoEncoder regularizes attention using human pixel masks on under 3% of training data. DeepfakeMAE trains a masked autoencoder only on real faces and flags faces whose parts reconstruct poorly. It reports 82.0% AUC on WildDeepfake against 66.2% for Xception. Neither uses within-image contrast. DeepfakeMAE avoids fake-data shortcuts entirely, which makes it a strong reference for whether training on fakes can be made to transfer at all.

Research Program

The detector is a patch-level scorer on a standard backbone. For an image we score every window $w$ in a multi-scale sliding set, giving a fake score $s_w \in [0,1]$. A pixel $u$ receives the average score of the windows covering it:

$$ M(u) = \frac{1}{|W_\lambda(u)|} \sum_{w \in W_\lambda(u)} s_w , $$

where $W_\lambda(u)$ is the set of windows containing $u$ at the current scale schedule $\lambda$. Early in training, $\lambda$ admits only large windows; small windows are added over time, following C-MIL's continuation idea. The image score is a softmax-weighted sum of window scores, following S-MIL, and is trained with the image label.

For each fake training image, we threshold the map $M$ at a high and a low quantile. Patches above the high threshold become pseudo-fake, patches below the low threshold become within-image pseudo-real, and patches in between are ignored. The full objective is

$$ \mathcal{L} = \mathcal{L}_{\text{img}} + \alpha\,\mathcal{L}_{\text{distill}}(s, \hat{M}) + \beta\,\mathcal{L}_{\text{within}} , $$

where $\mathcal{L}_{\text{img}}$ is image-level cross-entropy. $\mathcal{L}_{\text{distill}}$ trains a dense prediction head to match the thresholded map $\hat{M}$, which is the self-distillation step from SLV-SD. $\mathcal{L}_{\text{within}}$ is a contrastive loss that pulls pseudo-fake patch features apart from pseudo-real patch features of the same image. The weights $\alpha$ and $\beta$ start at zero and are ramped up once the map stabilizes. Patches from real images remain ordinary negatives throughout.

One condition limits where this can work. Within-image negatives are valid only when the unedited region is truly untouched. That holds for pixel-space inpainting (Repaint) and roughly for FaceForensics++ face swaps, where the background is original. It fails for latent-diffusion inpainting, where the whole image passes through the decoder. We therefore predict no gain on Repaint-LDM, and we treat a gain there as a sign that the mechanism is something other than what we claim.

Baselines

The naive reference point is an Xception binary classifier trained on the same data with image labels. On FaceForensics++ it reaches 99.26% on raw data (FaceForensics++) and 66.2% AUC on WildDeepfake as reported by DeepfakeMAE. For localization, its GradCAM map is the naive reference.

The strongest weakly supervised localization competitor is the Patches method from the diffusion-localization study. It reports 37.7% IoU and 79.8% pixel binary classification accuracy (PBCA) under image-level labels on Repaint-P2/CelebA-HQ. It also defines the ceiling: 83.7% IoU and 84.5% PBCA with full masks. We rerun it on the same backbone, data splits, and epoch budget. Beating it means a higher IoU on Repaint-P2 and a smaller drop from CelebA-HQ to FFHQ.

The strongest cross-dataset competitors are PRLE and DeepfakeMAE. For PRLE, the file gives only relative gains: about 6% average AUC across backbones, +9% Xception accuracy on Celeb-DF, and almost +15% EfficientNet AUC on DeeperForensics-1.0. No absolute AUC for PRLE on Celeb-DF is reported on Lacuna, so we reimplement it with the same backbone and a model zoo trained on our splits. For DeepfakeMAE, the target is the reported 82.0% AUC on WildDeepfake when trained on FaceForensics++. The RL test-time augmentation method (On Improving Cross-dataset Generalization of Deepfake Detectors) reports 0.669 AUC on Celeb-DF and 0.952 on DeeperForensics-1.0, and serves as the inference-time reference. All methods share the same training data, the same backbone where the method allows it, and an equal hyperparameter search budget of 20 trials tuned on a held-out FaceForensics++ manipulation, never on the test datasets. Beating the field means exceeding the Xception and PRLE reruns on Celeb-DF and DFDC and matching or exceeding 82.0% on WildDeepfake.

Three ablations isolate the key ingredient. The first, "foreign negatives," keeps everything identical but replaces within-image pseudo-real patches with patches sampled from other real images. The second, "GradCAM pseudo-masks," builds the pseudo-masks from GradCAM instead of voting. The third, "no continuation," uses the full window set from the start.

Why not just…?

Why not just mask the dominant region, as PRLE or random erasing does? Masking prevents the detector from relying on one region, but the model is still trained to tell whole fake images from whole real images, so dataset-level confounds such as compression and identity remain usable. The foreign-negatives ablation tests this directly. If PRLE or random erasing matches our method on Celeb-DF, our hypothesis loses its distinctive part, and we would report it as such.

Why not just use landmark-based face regions as the pseudo-mask? For FaceForensics++ face swaps, the edited area is roughly the inner face, so the inner face against the background is a cheap within-image contrast. This is a real threat on face-swap data, and we run it as a baseline. It cannot handle the 125k diffusion benchmark, where edits target eyes, mouth, nose, or attributes at varying locations. It is also the part-heuristic design that the parts-based study found brittle on DFDC. We expect learned voting to win there and possibly only tie on FaceForensics++.

Why not just spend a small mask budget, as the Locality-aware AutoEncoder does? That method gains 6.52% on unseen FaceSwap with under 3% annotated masks, and we include a 3%-masks variant of our own pipeline as an upper reference. The argument for pure weak labels is that new generators arrive faster than annotators can mask subtle edits; humans reach only about 58% accuracy on compressed manipulations (FaceForensics++). If 3% masks closes most of the gap, that is a useful finding, but it does not answer the weak-label question.

Why not just train only on real faces, as DeepfakeMAE does? One-class training sidesteps fake-specific shortcuts, and its WildDeepfake number is the bar we must clear. It does not output a localization, and it relies on hand-chosen part masks. If it still beats us cross-dataset, the conclusion is that within-image contrast reduces but does not remove shortcut learning. The paper would report exactly that.

Proposed Experiments

Localization experiments use the 125k diffusion benchmark under the image-level-label setup. We train on Repaint-P2/CelebA-HQ and report IoU, PBCA, and average precision in-distribution. We then test transfer to FFHQ and to the Repaint-LDM, LaMa, and Pluralistic generators. Detection experiments train on FaceForensics++ (HQ and LQ compression) with video labels applied to frames. We report frame- and video-level AUC on Celeb-DF v2, DFDC, DeeperForensics-1.0, and WildDeepfake, plus leave-one-manipulation-out transfer within FaceForensics++.

Three checks decide the claim. First, the foreign-negatives ablation must lose a clear share of the cross-dataset AUC gain, at least half; otherwise within-image contrast is not the cause. Second, across all methods and checkpoints, in-distribution localization coverage on the diffusion benchmark (IoU of the voted map) should predict cross-dataset AUC. A flat relationship would break the claimed link between the two gaps. Third, the predicted null on Repaint-LDM, where no pristine region exists, should hold. A gain there would mean the improvement comes from generic regularization rather than confound removal. If all three checks pass and we beat the Patches and PRLE reruns, the result supports the claim that part of the cross-dataset problem is a weak-supervision problem in disguise.