Untrustworthy Neighbors: Masked Pre-Training That Teaches Tiny Vision Transformers to Spot Splices

Face forgery detectors work well on the manipulation methods they were trained on and fail on new ones. One of the clearest fixes on record is to pre-train the detector without labels on real faces. Replacing ImageNet weights with face-specific masked image modeling lifts a ViT-Base detector's average cross-dataset AUC from about 61% to 86.56%. Masked image modeling means hiding most image patches and training the network to reconstruct them. That result comes from an 86M-parameter model, and the same recipe is known to break on small models. Lightweight vision transformers trained as masked autoencoders take a shortcut: they reconstruct each hidden patch by copying its nearest visible neighbor and never learn long-range structure. This failure is called "semantic aliasing."

We propose that this shortcut and the forgery-detection problem are the same problem seen from two sides. A small model fails at masked pre-training because it trusts adjacent patches too much. A forgery detector succeeds when it notices that adjacent patches should not be trusted, because they come from different sources across a blending boundary. We therefore make neighbors untrustworthy during pre-training. Some visible patches on real faces are covertly replaced with regions taken from other real faces or re-processed copies of the same face. The tiny model must reconstruct the hidden content and also say which visible patches are foreign. If the hypothesis holds, one objective removes the capacity bottleneck and trains the patch-consistency features that generalize across forgery methods. A ViT-Tiny detector could then be pre-trained on unlabeled real faces, with no teacher, and transfer from FaceForensics++ to Celeb-DF, DFDC and Celeb-DF++ at edge-device cost.

Core Claim

The existing fix for semantic aliasing in small masked autoencoders works against forensics. NoR-MAE (Closest Neighbors are Harmful for Lightweight Masked Auto-encoders) repels a visible patch from its closest neighbor patch with a decorrelation penalty. That penalty raises ViT-Tiny ImageNet-1K accuracy from 66.6% to 70.24%. But forgery generalization rests on comparing neighbors, and both of the following papers detect fakes as breaks in patch-to-patch agreement:

Blanket decorrelation of neighbors should shrink the contrast between "same-source neighbors" and "different-source neighbors," which is exactly the contrast a detector needs.

Our claim is that the shortcut should be made unreliable rather than penalized. We train with a pretext task in which the model must judge whether its neighbors are trustworthy. We predict three things. On faces, this gives a better lightweight initialization for cross-dataset forgery detection than vanilla MAE, than NoR-MAE, and than face-SSL distillation from a large teacher at equal compute. It does so by increasing the feature contrast across blending boundaries. It also fixes aliasing at least as well as neighbor repulsion, measured by reconstruction and linear-probe quality on faces.

How This Sits Against Prior Work

The motivating evidence comes from Revisiting Face Forgery Detection: From Facial Representation to Forgery Detection.

  • What it does. It pre-trains a ViT-Base with BEiT v2 on real faces, then fine-tunes two unshared branches under a decorrelation loss.
  • What it shows. Face-domain self-supervision is the large lever: about 61% to 86.56% average cross-dataset AUC from the backbone change alone, and 93.35% for the full CFB++ system.
  • How we differ. It never leaves ViT-Base, and its dual branch doubles inference cost. We keep the face-domain self-supervision lever but target a single ViT-Tiny.

Two other face-SSL detectors bound the space from the heavy side:

Our closest neighbor in mechanism is UniForensics' use of self-blending. Two things separate us from it:

  • It uses blends to form video-level positive pairs for a contrastive loss on a heavy encoder. We use foreign patches inside a masked reconstruction objective, at patch granularity, on single frames, to remove a capacity-specific shortcut in a tiny model.
  • The audio-visual approach needs audio and a frozen 3D front end. Ours needs only still real faces.

The method supplies come from both directions. From SSL scalability, we reuse the masked autoencoder and take NoR-MAE's diagnosis of semantic aliasing as the problem to solve. We also borrow the closest-neighbor-patch similarity as a diagnostic. From forensics, the PCL paper's Inconsistency Image Generator shows that synthetic splicing of real faces is a useful training signal. PCL uses it with supervised consistency labels during detector training. We use the same kind of signal as a label-free pretext during pre-training, combined with reconstruction.

The constraining evidence comes from two papers:

Research Program

The backbone is ViT-Tiny with a lightweight MAE decoder. Pre-training uses aligned, cropped real face frames from VoxCeleb2. We choose VoxCeleb2 because UniForensics already uses it as an unlabeled real-face source. Celeb-DF and DFDC real videos are excluded so that no test identities leak into pre-training.

Corrupting the visible patches. For each image we mask a high fraction of patches (75%, swept to 90%). Among the visible patches, a fraction $\rho$ is corrupted, drawn from one of three sources:

  • a region from a different aligned real face, color-matched to the host;
  • the same region from the same face after resampling or recompression at a different quality, which imitates the compression mismatch that PCL exploits;
  • nothing (the patch is clean).

The corrupted regions are soft-edged and elastically shaped, and are not aligned to the patch grid. This prevents the model from learning our patch-grid cut as a shortcut, which is the same overfitting the detectors in the forensics literature suffer from.

Objective. The loss is

$$ L = \frac{1}{|M|}\sum_{i \in M} \lVert \hat{x}_i - x_i \rVert_2^2 \;+\; \lambda \frac{1}{|V|}\sum_{j \in V} \mathrm{BCE}(\hat{s}_j, s_j), $$

with symbols defined as follows:

  • $M$ is the set of masked positions.
  • $x_i$ is the original uncorrupted pixel content at position $i$ and $\hat{x}_i$ its reconstruction.
  • $V$ is the set of visible positions.
  • $s_j$ is 1 if visible patch $j$ overlaps a foreign region and 0 otherwise.
  • $\hat{s}_j$ is a per-token prediction from a linear head on the encoder output.
  • $\lambda$ weights the two terms.

Reconstruction targets are always the original face. A model that copies a foreign neighbor into a masked slot is therefore penalized. This is the mechanism that is supposed to break aliasing without an explicit repulsion term.

Fine-tuning. After pre-training we fine-tune the full ViT-Tiny end to end as a binary real/fake classifier on FF++ (c23). We use one fixed recipe shared by every compared initialization, and no detector-specific modules.

Analysis. We measure two quantities on every pre-trained model:

  • Mean cosine similarity between each patch feature and its closest-neighbor-patch feature on clean real faces, which measures aliasing, following NoR-MAE's diagnosis.
  • "Boundary contrast" on held-out self-blended faces: the gap between neighbor similarity within a source and across a blend boundary.

The mechanism predicts the following ordering. NoR-MAE lowers both similarity numbers and therefore yields small boundary contrast. Vanilla MAE has high similarity everywhere, so contrast is also small. Our objective lowers reliance on neighbors while keeping contrast large. Cross-dataset AUC should track boundary contrast rather than ImageNet-style linear-probe accuracy.

Baselines

Naive reference: supervised ImageNet ViT-Tiny. This is ViT-Tiny initialized from supervised ImageNet weights and fine-tuned on FF++ with the shared recipe. It is the default a practitioner would use. Guo et al.'s 61% average for ImageNet-initialized ViT-Base suggests it will be weak. No reported result was found on Lacuna for ViT-Tiny on the FF++→Celeb-DF protocol, so we measure it ourselves. A random-initialization run is included as a floor.

Vanilla face MAE. This is ViT-Tiny pre-trained with standard MAE on the same VoxCeleb2 faces, for the same number of epochs and masking ratio. It is the direct small-model version of Guo et al.'s recipe and tests whether aliasing actually hurts on faces. Faces are highly structured, so the shortcut might matter less there than on ImageNet.

Face NoR-MAE: the ablation that isolates our key ingredient. This is ViT-Tiny pre-trained with the NoR-MAE neighbor-repelling loss on the same data and budget, with its published loss weighting and a matched tuning sweep. It is the existing treatment of the same capacity failure. Comparing it against ours separates "fix aliasing by decorrelation" from "fix aliasing by making neighbors unreliable." Two further ablations run under the same pre-training budget:

  • our objective with $\lambda = 0$, so corrupted patches are present but there is no detection head;
  • the detection head alone with no masking.

Strongest competitor: distilled face-SSL teacher. A face-pretrained ViT-Base is distilled into ViT-Tiny on the same unlabeled faces with CompRess-style similarity matching (CompRess: Self-Supervised Learning by Compressing Representations). We reproduce Guo et al.'s BEiT v2 recipe on our VoxCeleb2 frames, because no released checkpoint is listed in the file. CompRess and SEED (SEED: Self-supervised Distillation For Visual Representation) are the established cures for the SSL capacity bottleneck. SEED lifts MobileNet-V3 Large from 36.3% to 68.2%. This route inherits the representation that gave the 86.56% ViT-Base result. The student sees the same images and the same number of student updates. The teacher's pre-training compute is reported separately and charged against it in a compute-matched comparison.

Heavy reference points (not edge-deployable). These are reported from the papers, not rerun:

What counts as success. We count it as a win if the method beats face NoR-MAE and vanilla face MAE by at least 3 points of average cross-dataset AUC over three pre-training seeds. It must also match or beat the distilled student without using a teacher. The stretch result is a ViT-Tiny that reaches GenDF's 82.00% on Celeb-DF at a small fraction of ViT-Base inference FLOPs.

Why not just…?

Why not just add splicing augmentation during supervised fine-tuning? This is the most serious threat. PCL's generator and UniForensics' self-blending show that synthetic splices of real faces are useful, so the benefit might come from the splices alone. We test this directly. A supervised-ImageNet ViT-Tiny and a vanilla-face-MAE ViT-Tiny are both fine-tuned on FF++ plus the same foreign-region augmentation, with the corruption-detection loss added as an auxiliary target. If either matches our pre-trained model, the claim reduces to "splice augmentation works for small models." We would then report that and drop the pre-training story. Our reason for expecting a gap is capacity. A tiny model asked to learn consistency features from scratch during short fine-tuning on 1,000 FF++ identities has much less data than one pre-trained on VoxCeleb2-scale real faces. The experiment decides it.

Why not just distill the large face-SSL model into the tiny one? This could plausibly work as well. The CompRess and SEED results are large, and Guo et al.'s teacher is a demonstrably good face representation. We run it as the strongest baseline rather than dismissing it. There are two reasons ours could still be preferable:

  • The distilled student is bounded by a teacher that was itself trained with masked modeling and no consistency signal.
  • Distillation requires pre-training and hosting a ViT-Base, which is itself the heavy-dependency problem in this area.

If distillation wins on AUC, we test whether our pretext added on top of the distillation loss closes the remaining gap. That would reframe the contribution as a complement to distillation rather than a replacement.

Why not just keep a frozen ViT-Base and train adapters, as DeepFake-Adapter or GenDF do? Adapters reduce trainable parameters, not inference cost: the full ViT-Base runs on every frame. DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection also reports weakness on reenactment, where identity is unchanged. We do not claim to beat these methods on AUC. We report GenDF's 82.00% as the target to approach and give accuracy-versus-FLOPs and on-device latency so the trade is explicit.

Proposed Experiments and Decisive Checks

Main protocol. Train on FF++ c23. Test frame-level and video-level AUC on Celeb-DF v2, on DFDC, and on Celeb-DF++ split by face-swap, reenactment and talking-face scenarios using the released code OUC-VAS/Celeb-DF-PP. We also run leave-one-manipulation-out within FF++, with Face2Face held out as the reenactment case adapters handle poorly.

Compression. We repeat testing at FF++ c40 and at the Celeb-DF++ c45 setting, where that paper reports a 7.5% average drop. The purpose is to see whether the recompression variant of our corruption specifically buys robustness. That variant is ablated separately from the cross-face variant.

Cost. All models report parameters, FLOPs and measured latency on a mobile CPU.

First decisive check: does the pretext alone detect fakes? A linear probe on the per-token corruption head, with no fake data at all, is applied to Celeb-DF frames. If its zero-shot AUC is near chance, the pretext is not learning anything forgery-relevant. Any gain after fine-tuning would then have to come from better initialization quality alone, and the boundary-contrast analysis would arbitrate.

Second decisive check: the mechanism ordering. Across all pre-trained models, we test whether cross-dataset AUC correlates more strongly with boundary contrast than with closest-neighbor similarity or face linear-probe accuracy. If NoR-MAE beats us while showing low boundary contrast, the central hypothesis is wrong: neighbor decorrelation does not hurt forensics. We would say so.

Third decisive check: the gap to the large model. The same pretext is run on ViT-Small and ViT-Base. If the advantage over vanilla face MAE disappears at ViT-Base, as the capacity-bottleneck account predicts, that supports the claim that the method specifically repairs a small-model failure. If the advantage persists, the pretext is a general forensic pretext rather than a capacity fix. Either outcome is reported.