Resolution-Free Masked Autoencoding of Real Faces for Cross-Dataset Deepfake Detection

Deepfake detectors that score near 99% on the dataset they were trained on often drop to near chance on the next dataset. Most explanations blame the detector for learning shortcuts. One confound has not been tested cleanly. Benchmarks differ in native face resolution, compression, and post-processing, and every strong detector first resizes each face to a fixed pixel grid such as 224×224. That resizing reshapes exactly the fine blending and upsampling traces the detector depends on.

We propose to train a masked autoencoder that treats a face as a continuous function rather than a pixel array. It is trained only on real faces, and the masks are defined in face coordinates, so a mask hides "the mouth region" whatever the pixel count. The model therefore never needs to resize anything. A forgery is flagged when two things go wrong together: the hidden face part cannot be reconstructed well, and reconstruction error stops falling as the model is shown denser samples of the image. The second signal comes from the operator view. A real face behaves like a smooth continuous field, so more samples should help. Generator artifacts are tied to the generator's own pixel grid, so they leave error that more samples do not remove.

Core Claim

We make two claims, and they can fail separately. The first is diagnostic. A measurable share of the cross-dataset gap, for example training on FaceForensics++ and testing on Celeb-DF or WildDeepfake, comes from mismatched resolution and compression rather than mismatched manipulation methods. That share should shrink when resolution is controlled. The second is methodological. A detector that is discretization-invariant, meaning it gives the same answer whatever sampling density it is fed, should close that share of the gap. We expect it to beat a patch-based masked autoencoder given the same real-face training and the same resolution augmentation. Its scoring rule should not need the meta-learned mapping network used by the closest prior detector.

The detector uses a masked functional autoencoder: an encoder that reads an unordered set of (coordinate, color) samples, and a decoder that can be queried at any coordinate. The anomaly score combines anatomical reconstruction error with an "error-versus-density profile". The profile is how reconstruction error on a held-out face region changes as the visible sampling density grows. The profile is the new ingredient. Neither existing masked forgery detectors nor function-space autoencoders compute it.

Positioning Against Prior Work

The closest detector is DeepfakeMAE (Mover). It trains a ViT masked autoencoder on real faces only, masks whole facial parts (eyes, nose and cheeks, lips), and uses failure to reconstruct as a forgery cue through a meta-learned mapping network. Trained on FF++ and tested on WildDeepfake, it reports 82.0% AUC against 66.2% for Xception. This paper motivates our use of one-class anatomical masking, and we reuse that idea. Mover is locked to fixed patches and fixed input size, and it degrades when the masking ratio exceeds 85%. We differ in three ways: masks are defined in continuous face coordinates, no resizing is done at test time, and the score uses sampling density as a probe instead of a learned mapping network.

The closest operator-learning work is Autoencoders in Function Space. It builds autoencoders on coordinate-value sets with a mean-pooling encoder and a coordinate-MLP decoder. It trains by randomly dropping points, and reports stable zero-shot super-resolution up to 512× on PDE fields. It supplies our training principle: a mesh-agnostic encoder, a continuous decoder, and random point dropout. It has never touched images or forensics, its mean pooling cannot model where facial parts sit relative to each other, and its dropout is uniform rather than structured.

We take two further mechanisms from operator learning. From Universal Physics Transformers we take the Perceiver-style latent bottleneck, which keeps attention over tens of thousands of samples affordable. From CViT we take the Nadaraya-Watson continuous decoder, a decoder that predicts each query point as a kernel-weighted average over a learned latent grid. Its bandwidth β controls how sharp the reconstructions can be. CViT also warns about spectral bias, the tendency of coordinate networks to smooth fine detail. For a forensic model this is double-edged. Smoothing may wash out cues, but it may also leave generator artifacts conveniently in the residual. We treat β as an experimental variable rather than a fixed setting.

Two detectors constrain the design. Data-Independent Operator shows that frozen early filters remove content bias and reach 92.21% average accuracy on AIGCDetectBenchmark with 23.7M parameters. Despite the name, it is a fixed discrete convolution tied to one pixel grid. What Makes Fake Images Detectable? shows that small-receptive-field patch classifiers generalize across generators because local texture flaws transfer. Both imply that a strong competitor need not model global faces at all. Both also measure their cues in pixel units, which is the dependence we want to test.

FaceForensics++ gives the evidence that compression destroys high-frequency cues: XceptionNet falls from 99.26% accuracy on raw video to 81.00% on low-quality video. That warns us that a density profile built on high frequencies may vanish under heavy compression. The anatomical term is there to carry the signal in that regime.

Research Program

Diagnostic study. We first build a resolution-controlled version of the standard protocol, training on FF++ and testing on Celeb-DF and WildDeepfake. For every test face we record its native crop resolution. We then evaluate existing detectors in three ways:

  • on their usual resized inputs;
  • on test sets resampled so their native-resolution distribution matches FF++;
  • on FF++ test faces deliberately resampled to Celeb-DF's and WildDeepfake's resolution distributions and recompressed at FF++'s c23 and c40 levels.

If matching resolution recovers a meaningful part of the AUC lost across datasets, the confound is real. If it recovers almost nothing, the second claim loses its motivation, and we would report that as the main finding.

The model.

  • Input. Each face crop is mapped to normalized face coordinates by landmark alignment. Every pixel becomes a sample: its coordinate, encoded with Fourier features, plus its RGB value.
  • Masks. During training on real FF++ faces only, we hide one anatomical region, defined as a polygon in face coordinates, plus random extra regions. We also randomly subsample the visible points across a wide range of densities, so the model never sees a fixed pixel pitch.
  • Encoder. A small set of latent tokens cross-attends to the visible samples, then self-attends among themselves.
  • Decoder. A CViT-style kernel decoder predicts color at any queried coordinate.

Scoring a test face. We hide each anatomical region in turn. We reconstruct it from visible samples drawn at a ladder of densities, from sparse up to the image's full native resolution, and record the masked-region error at each rung. The result is an error-versus-density curve per region. A real face should give a curve that keeps falling toward the noise floor. A forgery should give a higher curve, a plateau, or both. The one-class score is the Mahalanobis distance, a distance that accounts for the spread and correlation of real-face curves, from the test curve to curves of held-out real FF++ faces. Training and scoring use no fake data.

Optional supervised variant. We will also report a variant that adds a light classifier trained on FF++ fakes on top of the frozen encoder. This makes the comparison with supervised detectors fair.

Baselines

Naive reference. The do-nothing reference is the standard XceptionNet pipeline: train on FF++ with bilinear resizing to 299×299, then test on Celeb-DF and WildDeepfake. Celeb-DF reports that nine such detectors, trained on FF-family data, average 56.9% AUC on Celeb-DF. DeepfakeMAE reports 66.2% AUC for Xception on WildDeepfake. We will retrain Xception ourselves on our splits so that every number we compare comes from one protocol.

Strongest existing method. On this exact question, the strongest competitor is Mover, because it is also a one-class masked autoencoder trained on real faces with anatomical masks. It reports 82.0% AUC on WildDeepfake when trained on FF++. We found no reported Mover AUC on Celeb-DF in the file. We will re-implement Mover on the same real FF++ training faces, with the same encoder parameter budget (within 10%), the same number of pretraining GPU-hours, and the same hyperparameter search budget. We will evaluate it both at its native fixed input size and on the resolution-swept test sets. Beating it means higher AUC on both WildDeepfake and Celeb-DF under the standard protocol. It also means a substantially smaller AUC drop across the resolution sweep, since that is where our claim predicts the advantage.

Strong cross-generator competitor. MDIO from Data-Independent Operator is a strong low-cost competitor whose generalization comes from frozen filters rather than learned representations. It reports 92.21% average accuracy on AIGCDetectBenchmark. That is a different setting (training on ProGAN, testing on whole generated images), so we run it on our face protocol with FF++ training. We also run our model on AIGCDetectBenchmark in its supervised variant to compare on DIO's own ground.

Resolution-agnostic local competitor. The truncated patch classifier of Chai et al. can run fully convolutionally at native resolution with no resizing. That makes it the right control for "just stop resizing".

Key ablation. The decisive ablation keeps everything about our model except discretization invariance. It is a patch-token ViT masked autoencoder with the same anatomical masks, the same one-class density-profile score (density emulated by downsampling before patching), and aggressive random-resize augmentation during training. If this ablation matches the full model, the operator formulation is not doing the work.

Further ablations. One drops the density profile and keeps only single-density reconstruction error. A second replaces anatomical masks with uniform point dropout, as in Autoencoders in Function Space. A third sweeps the decoder bandwidth β.

Why not just…?

Why not just train a standard patch MAE with heavy multi-resolution augmentation? This is the most serious threat. Resize augmentation may teach a discrete model enough scale robustness, and nothing in the file rules that out. CViT's result that 13M–92M parameter continuous models beat a 1B-parameter discrete model on Navier-Stokes (2.35% vs 2.78% relative error) concerns PDEs, not faces, so we do not lean on it. The key ablation above answers this question directly. If the augmented patch MAE matches the full model on the resolution sweep, we will say so. The contribution then reduces to the diagnostic study and to the density-profile score, which the ablation can also compute.

Why not just stop resizing and use a fully convolutional patch classifier at native resolution? It avoids interpolation, but its cue is measured in pixels, not in face-relative scale. A face filmed at half the resolution presents blending boundaries at half the pixel width. The resolution sweep tests whether this matters. Chai et al.'s classifier also needs fake training data, while our one-class model does not. We will report the supervised variant of our model specifically so this comparison is not lopsided.

Why not just use frozen filters as in DIO? DIO suppresses content bias cheaply. Its filters, however, respond to high frequencies, which low-quality compression removes, as the 99.26% to 81.00% XceptionNet drop on FaceForensics++ illustrates for high-frequency-reliant detectors. Our anatomical term is designed to survive compression. Running both methods at c23 and c40 across the resolution sweep will show whether that difference is real. If DIO matches us at c40, our compression argument fails.

Proposed Experiments and Decisive Checks

Data.

  • Training uses real FF++ faces only, at c23. FF++ fakes are used solely for the supervised variant and supervised baselines.
  • Testing uses FF++ held-out manipulations (Face2Face, FaceSwap, DeepFakes, NeuralTextures) at c0, c23, and c40, plus Celeb-DF, WildDeepfake, and DFDC Preview.
  • The cross-generator check uses AIGCDetectBenchmark.
  • Every test set is evaluated twice: once in standard form, and once in a resolution sweep covering 0.25× to 2× native scale, crossed with the three compression levels.

Metrics. We report frame-level AUC, plus video-level AUC where available. We also report the AUC drop from in-dataset to cross-dataset, and the slope of AUC across the resolution sweep, which directly measures discretization sensitivity.

First decisive check. The diagnostic: matching resolution should recover part of the Celeb-DF and WildDeepfake gap for Xception and Mover. If it recovers less than a few AUC points, the premise is weak, and we report that as the main finding.

Second decisive check. The full model should have a markedly flatter AUC-versus-resolution curve than both Mover and the augmented patch MAE. It should also be at least competitive with Mover's reported 82.0% AUC on WildDeepfake in the standard protocol.

Third decisive check. The density profile should add AUC over single-density reconstruction error on raw and c23 data. Its contribution should shrink at c40, while the anatomical term holds.

Mechanism check. On FF++ manipulations with known upsampling pipelines, the density at which the error curve plateaus should track the generator's native output resolution. If it does not, the profile's success, if any, needs a different explanation, and we will say so.