Frequency-Diverse Filter Banks for Cross-Dataset Deepfake Detection

Deepfake detectors often score above 99% on the dataset they were trained on and then fall to near chance on forgeries from a different source. One explanation is that they learn a shortcut. Many generators enlarge images with transposed convolutions, and these leave faint grid-shaped "checkerboard" patterns, which show up as sharp peaks at a few fixed spatial frequencies. A detector can separate real from fake almost perfectly by tuning itself to those peaks. Separately, recent work found that such detectors suffer rank collapse: their internal features shrink into a one- or two-dimensional subspace, and that collapse tracks poor generalization. No one has shown that these two observations describe the same event. This note proposes to test that link directly and to cut it at its source.

The tool is a property of the depthwise separable convolution, the building block of Xception, which is the standard face-forgery backbone. Such a layer first applies one small spatial filter to each channel separately and then mixes channels with a 1×1 projection, with no nonlinearity between the two steps. Each layer is therefore a linear filter bank whose frequency response can be read off exactly and cheaply with a Fourier transform of its tiny kernels. Our claim is that cross-dataset failure in convolutional detectors is preceded by a measurable frequency collapse: the early filter bank concentrates its gain on the training generator's upsampling frequencies. We propose a weight-space regularizer that keeps the bank's frequency response high-rank. It costs almost nothing to compute and is aimed at the shortcut's actual mechanism, unlike generic flattening of feature spectra.

Core Claim

The closest existing work is Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection (Effort). Effort observes that training loss on fakes drops to near zero almost immediately while loss on real images stalls, and that this coincides with feature rank collapse. It then freezes the principal singular directions of a CLIP ViT and tunes only an orthogonal residual. Effort treats the shortcut as an abstract residual subspace in a transformer and never asks what physical signal causes the collapse. Separately, CNN-Generated Images Are Surprisingly Easy to Spot... for Now shows that upsampling fingerprints are shared across CNN generators, and What Makes Fake Images Detectable? shows that truncated Xception patch models exploit local periodic textures. Neither paper looks at rank. Our proposal differs from both in two ways: it states a mechanistic, falsifiable link (harmonic concentration in the filter bank leads to feature collapse, which leads to cross-dataset failure), and it intervenes on the weights' frequency response rather than on feature statistics or a frozen subspace.

Concretely, consider a separable layer with $M$ input channels, depthwise kernels $k_m$, and pointwise matrix $W \in \mathbb{R}^{N \times M}$. For a grid of $\Omega$ frequencies $\omega$, the layer's response matrix is

$$R_{n\omega} = \sum_{m=1}^{M} W_{nm}\, \hat{k}_m(\omega),$$

where $\hat{k}_m(\omega)$ is the discrete Fourier transform of kernel $k_m$ zero-padded to the grid. This formula is exact only because Xception places no nonlinearity between the depthwise and pointwise steps (Xception). We use two quantities. The diagnostic is the harmonic gain fraction $h$: the share of $\|R\|_F^2$ that lies within a small neighborhood of the training generator's upsampling harmonics (multiples of $\pi/2$ for stride-2 upsampling chains). The regularizer adapts the eigenvalue-concentration penalty of Mitigating Spurious Features in Contrastive Learning with Spectral Regularization, but applies it to the frequency-response matrix instead of the feature covariance:

$$\mathcal{L} = \mathcal{L}_{\text{BCE}} + \beta \sum_{\ell \in \text{entry}} \sum_i \tilde{s}_{\ell,i}^2, \qquad \tilde{s}_{\ell,i} = \frac{s_{\ell,i}^2}{\sum_j s_{\ell,j}^2},$$

where $s_{\ell,i}$ are the singular values of $|R_\ell|$ for entry-flow layer $\ell$, $\mathcal{L}_{\text{BCE}}$ is the usual binary cross-entropy, and $\beta$ is a weight. We regularize only the entry-flow blocks. Patch-truncation results suggest the checkerboard signal is consumed early, and in deeper blocks frequency is measured on downsampled maps where its meaning becomes muddied.

The fast spectral-norm work in Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization supplies the computational idea: per-channel Fourier transforms of depthwise kernels are nearly free. That work bounds only the largest singular value of the operator, for stability. We reuse the Fourier trick to control the spread of the response across frequencies, which is a different goal.

Research Program

The first stage is diagnostic, and the regularizer only makes sense if it succeeds. We train Xception detectors under the standard FaceForensics++ protocol (FaceForensics++), and ProGAN-only detectors under the ForenSynths protocol. Throughout training we log four quantities: the harmonic gain fraction $h$ per entry layer, the effective rank of the response matrices, the effective rank of penultimate features, and the gap between fake and real loss that Effort calls asymmetry. The hypothesis predicts a time order. $h$ should spike and response rank should fall during the same early window in which fake loss collapses, and feature rank should fall after that. We then check whether the effect is causal. We remove harmonic-band energy from the training fakes by spectral inpainting of the peak bins, then retrain and check whether both the frequency collapse and the feature collapse disappear. If feature rank still collapses without any harmonic concentration, the mechanistic claim is wrong. The work then reduces to a negative result that constrains Effort's interpretation, and we would report it as such.

The second stage asks whether the separable structure matters. This addresses an honest worry. A standard convolution also has a computable frequency response, an $N \times M \times \Omega$ tensor rather than an $N \times \Omega$ matrix. So we run the same diagnostic and regularizer on a ResNet-50 by flattening its per-pair responses. If ResNet benefits equally, then depthwise separability is a convenience for computation and interpretation, not a necessary ingredient, and we would say so.

The third stage is the intervention. We sweep $\beta$ and compare a frequency-agnostic version of the regularizer, as written above, against a harmonic-targeted version that also penalizes $h$. The targeted version tests whether the benefit comes from general diversity or from denying this one shortcut.

Baselines

The naive reference point is Xception trained with plain binary cross-entropy on FF++, exactly as in FaceForensics++. That paper reports 99.26% accuracy on raw data and 81.00% on LQ (c40). Celeb-DF reports that FF++-trained detectors average 56.9% AUC on Celeb-DF, and that DSP-FWA is the most resilient at 87.4% average AUC across datasets. We found no Xception-specific Celeb-DF AUC on Lacuna, so we will measure it ourselves under a shared protocol. Beating the baseline means a gain in Celeb-DF and DFDC AUC averaged over at least three seeds, with in-domain FF++ AUC lowered by less than one point.

The augmentation baseline is the blur-plus-JPEG training recipe of CNN-Generated Images Are Surprisingly Easy to Spot... for Now. It reports 99.1% AP on StyleGAN2 and 88.2% AP on BigGAN from ProGAN-only training. It is the strongest cheap competitor on the GAN side because it also suppresses fragile high-frequency cues. We will run it on the same Xception backbone, with the same epochs and the same hyperparameter search budget, and we will also run it combined with our regularizer. To count as a win on ForenSynths, the regularizer must improve the average AP over the 11 generators on top of augmentation, not merely instead of it.

The receptive-field baseline is the Block-2-truncated Xception patch classifier of What Makes Fake Images Detectable?. The paper reports gains in Average Precision (AP) from PGAN to StyleGAN, but the file gives no numeric value. We will rerun it on our splits.

The key ablation is feature-space rank regularization: the same penalty from Mitigating Spurious Features in Contrastive Learning with Spectral Regularization applied to penultimate 2048-dimensional features, with an equal tuning budget for $\beta$. Its singular value decomposition on a batch is cheap at this size, so cost is not our argument. Our method wins only if it generalizes better than this ablation at matched in-domain accuracy.

The strongest existing method is Effort, which reports an average AUC of 0.940 across FaceForensics++, Celeb-DF-v2 and DFDC in Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection. Effort uses a pretrained CLIP ViT, so a raw comparison mixes up pretraining scale with method. We will report Effort under our evaluation protocol as an external ceiling. We will also report an ImageNet-pretrained Xception with our regularizer at roughly 23M parameters. We do not expect a CNN to beat Effort outright. The claims we can test are narrower: our regularizer should close a substantial fraction of the gap between baseline Xception and Effort, and the frequency diagnostic should explain which test sets each method fails on.

Why not just…?

Why not just use heavy blur and JPEG augmentation? It works well across GANs, but its own authors report chance-level performance on shallow manipulations and difficulty with super-resolution models, because blurring removes the high-frequency detail those methods alter. Augmentation also does not prevent the Celeb-DF cliff. It removes signal from the input, whereas our regularizer leaves the input intact and prevents the detector from betting everything on a few frequencies. The combined run in our Baselines settles whether the two methods are complementary or redundant. If augmentation alone already drives $h$ to the same level our regularizer reaches, our method adds nothing on GAN benchmarks, and we would report that.

Why not just apply the feature-covariance regularizer at the last layer? This is the most serious threat. It is cheap, it is generic, and it directly targets the collapse that Effort documents. Our argument is that flattening features after the fact can redistribute variance among features that all descend from the same harmonic detectors. Constraining the filter bank removes the shortcut upstream. The matched ablation is the deciding test. If feature-space regularization equals ours on Celeb-DF, DFDC and ForenSynths, the weight-space mechanism is unnecessary, and the contribution reduces to the diagnostic.

Why not just notch-filter the upsampling harmonics out of the input? If the shortcut is a handful of frequency peaks, deleting them at the input is simpler still. Two things argue against this, though neither is decisive. First, it needs the harmonic locations of every future generator. Second, FaceForensics++ shows that handcrafted frequency pipelines are fragile under compression. We include input notching as a baseline. If it matches our method, the honest conclusion is that the input, not the architecture, is the right place to intervene.

Why not just use Effort? For transformer detectors, Effort may well be the better choice. Our proposal targets convolutional detectors, which still dominate compressed-video forensics, and it tests a mechanism that Effort leaves unexplained. We treat Effort as the ceiling, not as a competitor we must beat.

Decisive Checks

Two results would make this proposal worthwhile, and two would kill it. The first supporting result: across checkpoints and seeds, the harmonic gain fraction measured on training data predicts FF++→Celeb-DF and FF++→DFDC AUC better than penultimate feature effective rank does. The second: the regularized Xception beats the feature-space ablation and input notching on those transfers, and on the ProGAN-to-11-generator average AP, at equal tuning budget. Testing at the c23 and c40 compression levels shows whether the gain survives codecs.

ProGAN→LDM/Glide transfer is the honest stress test. Diffusion decoders carry weaker checkerboard periodicity, so the mechanism predicts a smaller gain there. A large gain on diffusion images would mean the regularizer helps for some reason other than the one we claim. A secondary probe on the RAID benchmark checks whether a frequency-diverse filter bank also makes the detector harder to fool with adversarial perturbations.

The proposal fails if feature collapse persists after harmonic energy is removed from training fakes, or if the frequency-agnostic and harmonic-targeted regularizers perform no differently from last-layer feature flattening.