Spectral Right-Shift on Motion-Tracked Patch Graphs for Cross-Dataset Deepfake Detection

Deepfake detectors score near-perfectly on the dataset they were trained on, then fall toward chance on videos made by other generators. The usual explanation is that they memorize generator-specific fingerprints instead of learning what makes a forgery a forgery. Graph anomaly detection offers a different, generator-agnostic description of "abnormal." Normal data is smooth over a graph that connects things that should agree, and anomalies push the signal's energy toward the graph's high frequencies. This shift is called the "spectral right-shift," because energy moves toward the large end of the frequency axis. We propose to describe a face video as a graph whose edges link image patches that physically correspond: neighbors within a frame, and the same surface point tracked across frames by optical flow. On this graph, an authentic video's local image statistics should vary smoothly. A forged face, which is synthesized or blended inside an otherwise authentic frame, should break that smoothness wherever generated pixels meet real ones, in space or in time, whatever generator produced them.

The second ingredient addresses why frequency-based detectors fail under video compression. Compression mostly removes the very highest frequencies. If forgery energy lands in a middle-to-high band of the patch graph, a band-pass graph filter can measure it while ignoring the band that compression erases. A plain high-pass filter cannot do this. We will build the graph, check whether the right-shift exists on real versus fake faces, pick spectral bands using only re-compressed real videos, and test whether a small detector on these band energies, trained on FaceForensics++, transfers to Celeb-DF, DFDC, DFD and Celeb-DF++.

Core Claim

For a forged video, the energy of local forensic features measured on a motion-tracked spatio-temporal patch graph moves into a mid-to-high frequency band. That band can be identified from real videos alone, as the band whose energy stays stable across compression levels. Band energies in that region transfer across datasets better than pixel-grid Fourier or cosine-transform features, and they degrade less under heavy compression.

Write $X \in \mathbb{R}^{n \times d}$ for the $d$-dimensional feature of each of the $n$ patch nodes, and $L$ for the normalized graph Laplacian, a matrix whose eigenvalues act as graph frequencies between 0 (smooth) and 2 (rapidly alternating). Write $W_k$ for the $k$-th Beta-wavelet band-pass filter, a polynomial in $L$ that needs no eigendecomposition. The video is then described by its band-energy profile

$$E_k(X) = \frac{\lVert W_k X \rVert_F^2}{\lVert X \rVert_F^2}, \qquad k = 0, \dots, K,$$

where $\lVert \cdot \rVert_F$ is the Frobenius norm. The Rayleigh quotient $\mathrm{tr}(X^\top L X)/\mathrm{tr}(X^\top X)$ is a one-number summary of the same profile. The claim predicts three things: $E_k$ for fakes exceeds $E_k$ for reals in some band $k^\*$ below the top band; that gap holds across generators; and real-video $E_{k^\*}$ is roughly unchanged between lightly compressed (c23) and heavily compressed (c40) versions.

Relation to Prior Work

Three graph-anomaly papers motivate the mechanism and supply the tools. Rethinking Graph Neural Networks for Anomaly Detection identified the right-shift on fraud graphs and introduced Beta wavelets as band-pass filters, the BWGNN model. Rayleigh Quotient Graph Neural Networks for Graph-level Anomaly Detection showed that whole graphs can be separated by their accumulated high-frequency energy, and that this separation survives 15–20% random edge rewiring. Can Abnormality be Detected by Graph Neural Networks? showed that ordinary neighborhood-averaging graph networks act as low-pass filters that blur anomalies away. We reuse the Beta-wavelet filter bank and the graph-level energy summary. We differ in the setting: our graphs come from video geometry, and the anomaly is a region-level inconsistency rather than a deviant node attribute.

The closest applied works bracket the idea from two sides, and neither does it. Hyperspectral anomaly detection leveraging spatial attention and right-shifted spectral energy already moved BWGNN onto images. It uses nearest-neighbor pixel graphs on remote-sensing data with no generators, no cross-dataset shift and no compression. Multimodal Graph Learning for Deepfake Detection builds landmark graphs and frame-level temporal graphs for deepfakes, reaching 0.888 AUC on Celeb-DF when trained on FF++. It processes those graphs with spatial message passing, the low-pass operation the anomaly literature warns against, and runs its frequency analysis separately as a pixel cosine transform. The obvious combination is to swap BWGNN into that model's landmark graph, and we treat it as a baseline rather than the proposal. A graph of 68 landmark coordinates throws away the texture and blending evidence where most forgeries live. Its signal also depends on a landmark detector whose jitter rises under compression, so its "high frequencies" mix forgery with tracker noise.

Pixel-grid frequency detectors constrain the design. FreqNet uses fixed Fourier high-pass branches. FreqBlender argues directly that fixed high-pass filters miss forgery traces spread across bands, and builds pseudo-fakes from mid-band "structural" cosine components. Decoupling Forgery Semantics learns an adaptive high-pass filter. Together they support the band-selection premise. Their transforms, however, run on a fixed pixel grid and cannot follow a moving face. That gap is what the flow-tracked temporal edges fill. Celeb-DF++ supplies the main constraint: heavy compression (c45) cuts video-level AUC by 7.5% on average. DeepfakeBench adds that augmentations which blur away high frequencies trade quality-robustness for clean-video accuracy. Our compression-matched band selection is designed to avoid that trade.

Research Program

Graph construction. We tile the face crop, plus a ring of surrounding background, into 16×16 patches over a clip of 16–32 frames. Spatial edges join 4-neighbor patches within a frame. Temporal edges join each patch to the patch its center reaches in the next frame under optical flow computed on the full frame. Each temporal edge is weighted by flow forward–backward consistency, so poorly tracked points contribute little.

Node signals. We study two kinds and report both. The first is low-level residual statistics: per-patch variance and kurtosis of a fixed high-pass residual, plus 8×8 block-boundary energy. The second is frozen self-supervised patch tokens, reduced to a low dimension with principal component analysis. Comparing them tells us whether the right-shift lives in noise statistics, in semantics, or in both.

Stage 1: does the shift exist? On FF++ real and fake clips, we compute the full eigenvalue-binned energy spectrum exactly on small clips, then with the Beta-wavelet bank. We look for a fake-minus-real gap and locate the band. We check whether it appears for each of the four FF++ manipulation types separately and at c23 and c40. We also compare the face region against the background ring of the same video, where real and fake should match. If no consistent gap exists below the top band, the project stops at a negative diagnostic result, which is still worth reporting.

Stage 2: compression-matched band selection. Using only real videos re-encoded at several compression levels, we choose wavelet bands whose real-video energy stays stable across encodings. No fake labels are used in this step.

Stage 3: detection. Two detectors use the selected band energies, together with the face-minus-background contrast of each energy. One is a small supervised head trained on FF++. The other is a one-class density model fit to real videos only. We also train a learnable Beta-wavelet network end to end on the same graph, initialized from the selected bands.

Baselines

The naive reference is Xception trained on FF++ under DeepfakeBench's standardized augmentation. That benchmark found such "naive" convolutional detectors match many specialized ones under identical protocols, and the Celeb-DF paper reports detectors averaging 56.9% AUC on Celeb-DF. Beating it is necessary but not sufficient.

The strongest existing competitors on this exact protocol are FreqBlender and FakeRadar. FreqBlender reports 94.59% AUC on Celeb-DF and 74.59% on DFDC when trained on FF++ (FreqBlender). FakeRadar, which probes for outliers on a frozen CLIP backbone, reports 84.1% AUC on DFDC (FakeRadar). On Celeb-DF++, the reported FF++-trained average is 73.8% video-level AUC across evaluated detectors. We found no reported result on Lacuna for FreqBlender or FakeRadar specifically on Celeb-DF++, so we will run both ourselves in DeepfakeBench with the authors' settings. We also include SPSL, DeepfakeBench's most cross-domain-robust detector at 78.75% average AUC, and the multimodal graph model at its reported 0.888 Celeb-DF AUC.

Fairness rules apply to every method. All train on FF++ c23 with identical face crops, frame sampling and augmentation, and all receive the same hyperparameter search budget, tuned only on an FF++ validation split. Because our head is small, we will also report it fused with the Xception baseline, at matched compute. Beating the strongest method means one of two things. Either the band-energy detector, alone or fused, matches FreqBlender within one AUC point on Celeb-DF while exceeding FakeRadar on DFDC. Or, the more likely and more interesting outcome, it loses less than half of FreqBlender's AUC drop on Celeb-DF++ at c45 and on the talking-face subset, where face-swap-trained detectors are expected to fail.

Three ablations isolate the key ingredients. Removing temporal edges leaves a per-frame grid graph, which tests whether motion tracking matters. Replacing the selected band with the top band only, a plain Laplacian high-pass, tests band selection. Replacing the patch graph with the landmark graph tests the choice of nodes.

Why not just…?

Why not just use motion-compensated frame differencing? Taking the temporal edges alone with a pure high-pass filter is essentially warping each frame onto the next and measuring the residual. That needs no graph theory, and it is a serious threat. It differs from our proposal in two ways: it is fixed to the highest frequency, which compression attacks most, and it ignores coupling between spatial and temporal edges. The top-band and temporal-only ablations run exactly this baseline. If it matches the band-pass detector at c40 and c45, the honest conclusion is that the graph framing is a useful diagnostic, not a better detector, and we will say so.

Why not just put BWGNN on the landmark graph of the multimodal graph model? That is the obvious intersection, and we run it. We expect it to underperform for two reasons. Landmark coordinates carry no texture or blending evidence. The landmark detector's own compression-induced jitter is high-frequency noise of exactly the kind a high-pass filter amplifies. Stage 1 measures that jitter directly on re-compressed real videos.

Why not just use a pixel-grid Fourier or cosine band-pass, as FreqBlender's mid-band analysis does? A 2D transform on a fixed crop treats a face that moves between frames as changing content. It cannot ask whether the same tracked surface point keeps consistent statistics over time. If the per-frame grid ablation equals the full graph, this objection wins. In that case the contribution narrows to compression-matched band selection, which can be ported back to pixel transforms.

Why not just fine-tune a large frozen backbone with outlier probing, as FakeRadar does? That approach is strong on DFDC, but its decision boundary lives in a semantic feature space fit to known fakes. Our real-only density variant tests whether a label-free energy statistic transfers to Celeb-DF++ talking faces, where FakeRadar's clusters were never shaped. Fusing the two tests whether they are complementary.

Decisive Checks

Four results decide the proposal. First, Stage 1 must show a fake-minus-real energy gap below the top band, present for at least three of the four FF++ manipulations, and absent in the face-versus-background comparison for real videos. Second, the band chosen without fake labels must coincide with, or overlap, the band that best separates fakes from reals. This is the central falsifiable prediction. Third, on Celeb-DF++, comparing c23 to c45, the band-energy detector should lose clearly less AUC than the reported 7.5% average drop and than our rerun of FreqBlender. Fourth, the temporal-edge and band-selection ablations must each cost AUC; otherwise the simpler alternatives above stand. AUC is reported at both frame and video level on Celeb-DF, DFDC, DFD and the three Celeb-DF++ evaluation tracks, with three seeds and bootstrap confidence intervals.