Large self-supervised speech models, such as wav2vec 2.0, XLS-R and WavLM, learn speech representations from unlabeled audio and are now the standard front end for detecting synthetic speech. They pose a problem for that use, though. Their deeper layers are trained to ignore exactly what a forensic detector needs: vocoder glitches, phase errors, jitter and shimmer, and the seams where a synthetic phrase has been spliced into real speech. Current detectors cope by picking or weighting layers, giving each layer one importance number. We propose instead to treat the model's hidden states as a grid of nodes, one per (layer, time frame), and to run attention only along a small set of typed local connections. Those connections are "same layer, next frame", "same frame, next layer up" and "diagonal neighbor". Each connection type gets its own learned bias. The detector can then follow a trace such as "a shallow-layer discontinuity at frame $t$ that coincides with a mid-layer disfluency nearby", which layer weighting cannot express.
The design borrows one specific lesson from graph learning. A controlled study of relation-aware transformers on knowledge graphs, What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs?, found three things. Restricting attention to graph neighbors carried most of the benefit. Relation biases without that restriction made results worse than a plain transformer. The restricted model also degraded far less when relation types were held out. We bet that the same holds for audio. In this setting "held-out relations" correspond to unseen generators, and the regime where the bet matters most is partial forgeries, where current detectors fail badly.
Core Claim
A detector that attends over a sparse, typed layer–time graph built from a frozen or lightly tuned self-supervised speech model will beat additive layer aggregation and fixed hierarchical attention in two places. The first is cross-generator generalization on In-the-Wild and ASVspoof 2021 DF. The second, and larger, is localizing spliced synthetic segments on Speech-Forensics. We further claim that the gain comes from the sparsity pattern, not from extra capacity. A dense transformer over the same layer–time tokens, with the same relation biases but no mask, should do worse, especially on held-out generators. If the dense version matches the sparse one, the graph framing is unnecessary and the honest conclusion is "use a transformer over all hidden states". The experiments are built to detect that outcome.
What Prior Work Does and Does Not Do
Three lines of work motivate the gap. Layer-probing studies establish where artifacts live. Comprehensive Layer-wise Analysis of SSL Models for Audio Deepfake Detection and El Kheir et al. locate detection capacity in roughly layers 4–12. Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification finds that XLSR-53 layer 5 alone is the best single layer. Salvi et al. report that detection plateaus or declines at the largest model scales. All three treat layers as things to keep or drop, not as states that interact.
Two aggregation methods go further but stay non-relational. Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection weights all 24 WavLM layers with a squeeze-and-excitation block (a small network that outputs one gate per layer). That gives one global weight per layer, so the method cannot connect a frame in one layer to a nearby frame in another. HierCon: Hierarchical Contrastive Attention for Audio Deepfake Detection is the closest existing work. It runs temporal attention within each layer, then attention within fixed layer groups (1–8, 9–16, 17–24), then attention across groups, and adds a contrastive loss. Its temporal pooling happens inside each layer before any cross-layer step, so it cannot represent "frame $t$ in layer 3 relates to frame $t\pm1$ in layer 9". Its layer groups are also hand-fixed. Our model differs on both counts: cross-layer and cross-time interaction happen jointly at frame resolution, and the hand-chosen groups are replaced by typed local edges.
Graph attention has been used for anti-spoofing, but over a single CNN output rather than a layer hierarchy. AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks builds a graph with two node types (frequency sub-bands and time segments) over RawNet2 features. Graph Attention Networks for Anti-Spoofing uses separate fully connected spectral and temporal graphs. Neither has typed edge biases or sparse neighbor masks, and neither sees self-supervised layers.
Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis plays two roles. It constrains the design, because it shows that AASIST and RawNet2 exceed 24% EER on partial forgeries. It also supplies the strongest localization competitor, the TEST network, which uses difference convolutions and multi-scale regression on WavLM features.
On the method side, we reuse Graphormer's idea of additive attention biases indexed by structural relation, from Do Transformers Really Perform Badly for Graph Representation?. We take the "mask first, then bias" ordering from the knowledge-graph study above, which is the paper that makes the dense alternative a negative prior rather than a neutral one.
The Model
Take a pretrained speech model with $L$ layers. Its hidden states $h_{\ell,t}$, for layer $\ell$ and frame $t$, become nodes after a shared projection plus a learned layer embedding. Each node connects to a fixed neighborhood $\mathcal{N}(\ell,t)$ made of three edge types:
- temporal edges to $(\ell, t\pm k)$ for lags $k \in \{1,2,4\}$;
- vertical edges to $(\ell\pm1, t)$;
- diagonal edges to $(\ell\pm1, t\pm1)$.
Each type $r$, together with its signed offset, indexes a scalar bias $b_r$. Attention is computed only within the neighborhood:
$$ \alpha_{(\ell,t),(\ell',t')} = \mathrm{softmax}_{(\ell',t') \in \mathcal{N}(\ell,t)}\!\left(\frac{q_{\ell,t}^\top k_{\ell',t'}}{\sqrt{d}} + b_{r(\ell,t,\ell',t')}\right), $$
where $q$ and $k$ are query and key projections and $d$ is their width. Each node also receives the difference $h_{\ell',t'} - h_{\ell,t}$ as an extra value input on temporal edges. The purpose is to make discontinuities, which is what splices are, cheap to represent.
After a few such layers, two heads read the graph. A frame-level head produces a forgery probability per frame for localization. An attention-pooled utterance head produces the spoof score. With 24 layers and about 200 frames, the graph has roughly 4,800 nodes but only about 14 neighbors per node, so cost grows linearly with clip length rather than quadratically.
Research Program
The work has three parts. First, we build the model and train it on ASVspoof 2019 LA, from The ASVspoof 2019 database., for full-utterance detection, and on Speech-Forensics for partial forgeries. We use XLS-R and WavLM-Large backbones in two regimes, frozen and fine-tuned, so that comparisons with HierCon and TEST are like-for-like.
Second, we isolate the mechanism. One set of ablations swaps the mask (sparse vs. dense) while holding parameters fixed. Another removes edge types one at a time. A third removes the difference messages.
Third, we analyze what the graph learns. Per-edge-type attention mass is measured separately on authentic speech, fully synthetic speech and splice boundaries. Per-layer attention mass is compared against where the layer-probing literature says artifacts sit. The hypothesis to falsify is that boundary frames draw on vertical and diagonal edges into mid layers, while vocoder artifacts are handled by temporal edges in shallow layers. If both kinds of evidence are routed identically, the "relational routing" story is wrong even if the accuracy numbers improve.
Baselines
The naive reference point is a single intermediate layer with attentive statistics pooling (pooling that weights frames before taking mean and variance). Concretely this is XLSR-53 layer 5 with that pooling, as in Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification, which reports 0.31% EER on ASVspoof 2019 LA. It is the strongest result in the file on that benchmark. We retrain it on our data splits and also run it at the best layer for WavLM. Because 2019 LA is saturated, we do not expect to beat it meaningfully there. What matters is that it does not beat us on In-the-Wild, 2021 DF or Speech-Forensics.
The additive aggregation baselines are AttM (Attentive Merging, reported 0.65% EER on 2019 LA, 3.50% on 2021 LA and 3.19% on 2021 DF) and softmax scalar layer weighting over layers 1–12. Both are reimplemented on the same backbones with the same back-end width and tuning budget.
The strongest full-utterance competitor is HierCon (HierCon). It reports 1.93% EER on ASVspoof 2021 DF, a 36.6% relative gain over its XLS-R baseline, and 6.87% EER on In-the-Wild, a 22.5% gain over independent layer weighting. No reported 2021 LA number was found on Lacuna for HierCon. We will rerun HierCon in its own fine-tuned XLS-R regime, using the same training data (2019 LA), the same augmentation and an equal hyperparameter search budget (the same number of trials per method). Beating it means lower EER on both 2021 DF and In-the-Wild across three seeds, with non-overlapping seed ranges, using a back-end with no more parameters than HierCon's.
The strongest localization competitor is TEST from Speech-Forensics. It reports 83.55% average mAP, 1.07% segment-level EER and 92.19% segment F1, plus an utterance-level EER that falls to 0.64% with WavLM-Large. We compare against TEST on the WavLM-Large backbone and count a win only if segment F1 and mAP both improve.
The key ablation is a dense transformer over the identical layer–time tokens, with the same relation biases and parameter count but no mask. Its counterpart removes the biases and keeps the mask. Together these separate "where attention may go" from "what each edge type means".
Why not just…?
Why not just use the best single layer with attentive pooling? On 2019 LA it is very hard to beat at 0.31% EER. However, its authors note that it struggles on the WaveRNN attack A10. Speech-Forensics also shows that detectors built for full forgeries collapse on partial ones, which require context from higher layers. If our model fails to beat this baseline on Speech-Forensics segment F1 and on In-the-Wild, the multi-layer graph is not earning its cost and we will say so.
Why not just learn per-layer weights, as AttM does? This is cheap and strong. HierCon's 22.5% In-the-Wild gain over independent weighting already suggests that cross-layer structure matters. Even so, per-layer weights cannot express frame-to-frame, cross-layer evidence by construction. Comparing frame-level localization on spliced data, where weighting has no mechanism to mark a boundary, settles the question directly.
Why not just put a transformer over all layer–time tokens? This is the real threat. If dense attention matches sparse attention, our contribution reduces to "attend jointly over layers and time". The knowledge-graph evidence that unmasked relation biases fall below a plain transformer (2.1% vs. 2.7% on CWQ) and degrade much more on held-out relations (−29.2 vs. −7.2 points) predicts that the dense version will lose. That evidence comes from a different domain, however, so we test it directly through the matched dense ablation and the held-out-generator split described below.
Why not just fine-tune the backbone end to end, or fuse several systems? End-to-end fine-tuning lets the backbone itself un-learn its invariance. We therefore run every head on a fine-tuned backbone as well, and require our gains to persist there. Score fusion produced a 47% min t-DCF reduction in Graph Attention Networks for Anti-Spoofing, but at several times the compute. We report a fusion of the single-layer and AttM systems as a reference, and compare both accuracy and inference cost.
Proposed Experiments and Decisive Checks
All methods train on ASVspoof 2019 LA and are evaluated on 2019 LA, 2021 LA, 2021 DF and In-the-Wild, using EER and min t-DCF where defined. Separately, all methods train and test on Speech-Forensics, reporting mAP, segment EER, segment F1 and utterance EER.
The first decisive check is a held-out-generator protocol on Speech-Forensics. We train on a subset of its 11 TTS systems and test on the rest. This is the audio analogue of the held-out-relation test, and we predict that the gap between sparse and dense attention widens there.
The second check is an attack-wise breakdown on 2019 LA, focused on A10, to see whether cross-layer edges recover the attacks on which shallow-layer detectors fail.
The third check is the routing analysis described in the research program. Attention mass at annotated splice boundaries should shift toward vertical and diagonal edges into layers above roughly 8, relative to mass on fully synthetic frames.
The proposal fails cleanly if any of three things happens: the dense ablation matches the sparse model on held-out generators, AttM matches it on In-the-Wild, or TEST matches it on segment F1. Under any of these outcomes the graph structure is not the operative ingredient.
References
- What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs?
- Comprehensive Layer-wise Analysis of SSL Models for Audio Deepfake Detection
- Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification
- Comparative Analysis of ASR Methods for Speech Deepfake Detection
- Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection
- HierCon: Hierarchical Contrastive Attention for Audio Deepfake Detection
- AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks
- Graph Attention Networks for Anti-Spoofing
- Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis
- Do Transformers Really Perform Badly for Graph Representation?
- The ASVspoof 2019 database