Which Frequencies Carry the Lie? Band-Gated Facial Signals for Cross-Dataset Deception Detection

Deception detectors trained on one video corpus mostly fail on another. On a six-dataset benchmark spanning courtroom trials, game shows and lab studies, the best multi-source system reaches only about 59% accuracy when tested on a held-out dataset (Benchmarking Cross-Domain Audio-Visual Deception Detection). That is near chance for a balanced binary task. The usual explanation is that models latch onto whatever separates the training sets: camera, lighting, codec, framing. The signal they are meant to detect is involuntary "leakage," meaning the bodily traces a liar cannot fully suppress, such as fleeting facial twitches, pulse-driven color changes in the skin, and shifts in blinking and gaze.

This proposal starts from a simple physical observation. Leakage and nuisance live at different speeds. Pulse sits near 1 Hz. Micro-expressions and saccades flash over 100–500 ms. Thermal and postural drift unfold over tens of seconds (Your blush gives you away). Lighting drift is very slow, and compression artifacts are tied to specific frequency ranges. We propose to break per-face-region signal traces into temporal frequency bands and measure, band by band, how much each one predicts deception versus how much it predicts which dataset a clip came from. We then train a small detector that keeps the first kind of band and shuts off the second. In two sentences, the claim is this: deception cues that transfer across datasets occupy different temporal frequency bands than dataset-identifying nuisance, and a detector that learns to gate bands by that criterion should generalize better than broadband video backbones. The band-level map is useful by itself, because it says which leakage channels survive real-world recording.

Core Claim

"Dual-domain" feature extraction means processing signals both in their native layout (pixels or regions) and in a frequency representation obtained with a Fourier or cosine transform. It has worked well for forgery detection. D2Fusion combines spatial attention with cosine-transform spectral attention and gains 6.25% AUC when trained on FaceForensics++ and tested on another forgery dataset. Spatial-Phase Shallow Learning shows that phase spectra expose subtle facial manipulation traces. Both methods target synthetic artifacts, not human physiology, and both operate on spatial frequencies within frames.

Our claim concerns temporal frequency, measured per facial region, and is falsifiable in a specific way. If the bands that best predict dataset identity overlap heavily with the bands that best predict deception, band gating cannot help. We would then report that cross-dataset failure in this field is not a spectral-nuisance problem. If the bands separate, a detector restricted to deception-heavy, domain-light bands should beat the current 59.02% multi-to-single average. It should also degrade less when test videos are re-compressed.

Research Program

Input signals. For each clip, we track facial landmarks and define roughly two dozen regions of interest, following the landmark-region design of the r-PPG pipeline in Your blush gives you away. R-PPG, or remote photoplethysmography, recovers the pulse from tiny skin-color changes in ordinary video. From each region we extract three time series:

  • mean color, which carries pulse;
  • landmark and optical-flow motion magnitude, which carries twitches and over-control;
  • facial action-unit intensities, which carry expression.

All clips are resampled to a common frame rate. The analysis is capped below the lowest Nyquist frequency across datasets, so no band exists in one corpus and not another. Short-time Fourier transforms use Hann windows, which avoids the spectral-leakage problem the file flags for non-stationary signals.

Model. The model alternates two kinds of blocks, borrowing the alternating structure of AFIDAF. AFIDAF swaps between spatial convolutions and Fourier-domain channel masks inside an image backbone. In our version:

  • Spatial block. Mixes information across facial regions using a small graph over the face layout.
  • Frequency block. Multiplies each region-by-band coefficient by a learned gate.

We move the Fourier side from image space to the time axis of physiological traces, and we make the gates interpretable as bands. The model stays at well under a million parameters, because the largest labeled corpus is small (Real-Life Trial has 104 clips; Bag-of-Lies has 325 trials).

Training objective. Let $g_{b} \in [0,1]$ be the gate on band $b$, and let $D_b$ be the held-out accuracy of a linear probe that predicts source dataset from band $b$ alone, computed on training sources only. We minimize

$$ \mathcal{L} = \mathcal{L}_{\text{dec}} + \lambda \sum_{b} g_b \,\max(0, D_b - D_{\text{chance}}), $$

where $\mathcal{L}_{\text{dec}}$ is the deception cross-entropy, $D_{\text{chance}}$ is chance-level dataset accuracy, and $\lambda$ trades accuracy against suppressing domain-revealing bands. A second variant replaces the fixed penalty with band-wise gradient reversal against a dataset classifier. Gradient reversal is the adversarial alignment trick that Xia and Wang use to align macro- and micro-expression apex frames. Here we apply it per frequency band rather than per frame.

Band map. The analysis output is a map over (region, signal type, band) cells. Each cell records two quantities: deception predictability within a dataset, and dataset predictability across datasets. This map is the scientific deliverable whether or not the detector wins.

Role of prior work. Several papers motivate the design:

  • The cross-domain benchmark establishes the failure and shows that hand-picked visual features generalize better than audio spectrograms. This suggests the choice of signal representation matters more than the fusion module.
  • Inception Transformer argues that self-attention behaves like a low-pass filter. That is a reason to suspect large video transformers underweight brief, high-frequency leakage.
  • D2Fusion notes that lossy compression wipes out high-frequency cues. This constrains us: we cannot assume high bands survive courtroom video, so we measure that directly.
  • Your blush gives you away explicitly leaves the fast-pulse versus slow-thermal timing mismatch unresolved. Band decomposition handles that mismatch naturally.

Baselines

Do-nothing reference. A majority-class predictor gives the floor. A logistic regression on clip-level means and variances of the same region traces, with no frequency decomposition, tells us whether spectral structure adds anything beyond simple statistics.

Strongest cross-domain method. This is Attention-Mixer with Multimodal Inter-Domain Gradient Matching from Benchmarking Cross-Domain Audio-Visual Deception Detection. It reports 59.02% multi-to-single average accuracy, against 58.88% for Attention-Mixer alone. It is the right competitor because it is the only method in the file evaluated under exactly our protocol and designed for exactly our failure. We rerun it with its public settings on the same six datasets (Real-Life Trial, Bag-of-Lies, MU3D, Box of Lies, DOLOS, MDPE), using the same leave-one-dataset-out splits and the same hyperparameter search budget as our model. We run it in two forms: its native modalities, and visual-only on our region traces, so that input differences do not decide the comparison. Beating it means a higher leave-one-out average, with a paired bootstrap interval over target datasets that excludes zero. The file gives no per-target breakdown for this method; we found no reported result on Lacuna for its individual held-out-dataset accuracies.

Strongest within-dataset system. This is AFFAKT, which transfers facial-expression knowledge from large emotion corpora into VideoMAE features through hierarchical optimal transport. It reports visual F1 0.8760 and accuracy 0.8670 on Real-Life Trial, and F1 0.7102 on DOLOS. We do not expect to beat it in-domain with a tiny model. We run its VideoMAE features under the leave-one-dataset-out protocol to test whether its in-domain strength survives transfer.

Dense-trajectory pipeline. For in-domain context we also run Deception Detection in Videos, which reports AUC 0.877 on Real-Life Trial.

Key ablations. These isolate the gating ingredient:

  • Fixed-band. Hand-chosen physiological bands (pulse, micro-movement, slow drift) fed to the same classifier.
  • Ungated. The same model with all gates fixed open.
  • Unpenalized. The same model with learned gates but $\lambda = 0$.

All three share architecture, data and tuning budget with the full model.

Why not just…?

Why not just apply fixed physiological band-pass filters and a linear classifier? This is the real threat. It is how r-PPG pipelines already work, and with 100–300 clips per dataset, learned gates may add nothing. We include it as a first-class arm rather than a straw man. If fixed bands chosen by the same leave-one-dataset-out criterion match the learned model, we will say so. The contribution then reduces to the band map and the finding that band selection, not learned filtering, drives transfer. That finding would still change practice, because current cross-domain work does no band selection at all.

Why not just use a stronger pretrained video backbone such as VideoMAE with affective transfer? AFFAKT shows such backbones are strong in-domain. However, nothing in the file shows them transferring across datasets, and the low-pass argument of Inception Transformer suggests they may underweight brief leakage. We settle this empirically. We run AFFAKT-style features under leave-one-dataset-out, and we probe the temporal spectrum of those features to see whether they retain micro-movement-band energy.

Why not just add domain generalization or compression augmentation to existing features? Gradient matching is the strongest baseline above, and it moves accuracy only from 58.88% to 59.02%. That small gain suggests alignment applied after features are extracted cannot remove nuisance already baked into them. Compression augmentation is cheaper and plausible. We add it to every arm, including ours, so any gain from band gating must come on top of it.

Proposed Experiments

Main test. Leave-one-dataset-out across the six corpora, reporting per-target accuracy and AUC, the six-target average compared against 59.02%, and subject-disjoint splits within each source.

Decisive check 1: band separation. Before training any detector, we compute the band map on training sources and ask whether deception-predictive and dataset-predictive bands separate. A rank correlation near one between the two predictabilities across cells falsifies the core claim. We would publish that negative result instead.

Positive controls. These verify that the decomposition recovers what it claims to:

  • On the pyMMER data, the pulse-band color trace should track ECG heart rate. The original pipeline reports a 0.86 correlation for r-PPG against ECG, which is our reference.
  • On CASME II, the micro-movement band should carry most of the micro-expression signal. The temporal reference there is 79.1% accuracy from Xia and Wang, which encodes speed through playback-rate resampling rather than explicit bands.

Decisive check 2: compression stress test. We re-encode every target clip at several H.264 bitrates and plot accuracy against bitrate for every arm. If band gating works for the stated reason, its curve should be flatter than the ungated and broadband baselines.

Decisive check 3: courtroom confound. Real-Life Trial labels may correlate with trial context. When it is the target, we additionally report results restricted to clips with matched camera setups where possible. We treat any gain that vanishes under that restriction as a shortcut, not leakage.