Systems that detect fake or replayed speech have a known problem with silence. The quiet stretches before, between, and after words carry real physical evidence: the hiss of a loudspeaker, the echo of a playback room, the unnaturally clean zero-level output of a speech generator. They also carry bookkeeping accidents of how each dataset was assembled, such as how much leading silence the organizers happened to leave on genuine versus fake files. Deep detectors use both kinds of cue without distinguishing them. If silence is trimmed, accuracy drops. If it is kept, the detector memorizes the dataset. We propose to separate the two kinds of cue with a small set of hand-designed, physically interpretable measurements of the non-speech regions, grouped into families such as duration and placement, noise-floor level, noise-floor spectral shape, amplitude distribution near zero, and reverberant decay after speech ends. These measurements serve two purposes. First, they audit how much of a deep detector's decision each family explains. Second, the audit decides which families to randomize during training and which to leave alone.
The claim, in two sentences, is this. Blanket fixes for the silence shortcut, whether trimming silence or randomizing everything in it, throw away the channel physics that detection of replayed audio depends on. An explicit engineered decomposition of non-speech content lets us neutralize only the families that act as shortcuts, and this should hold up better than either blanket fix across both synthetic-speech and replay conditions.
Core Claim
The literature points both ways on silence, and both sides have evidence. On one side, trimming non-speech frames with voice activity detection (VAD, a model that marks which frames contain speech) consistently hurts. It hurt replay detection in Font et al.'s analysis of ASVspoof 2017 features. It hurt across both attack scenarios in the STC ASVspoof 2019 systems, in noisy conditions in the joint enhancement/anti-spoofing work of Wang et al., and on ASVspoof 2019 LA in Zhang et al.'s silence study, where trimming also worsened overfitting. On the other side, the ASVspoof 2021 overview reports that detectors trained on full utterances collapse on hidden test subsets with silence removed. Shim et al. show that loss on genuine speech falls to about 1e-4 almost immediately. On AASIST, MP3-compressing only the genuine files produces an error ratio of 137.33 relative to compressing only the spoofs. Their bias also persists after silence trimming.
Our reading is that "silence" is not one variable. Several papers already hint at the component variables. Lapidot and Bonastre show that spoofed audio has a sharper amplitude peak near zero, and that this difference is concentrated in non-speech regions. Remapping only the non-speech amplitude distribution raises the EER of a CQCC-based detector fourfold. Suthokumar et al. show that low-energy frames are more discriminative than voiced ones. They also note that mixing unvoiced consonants with pure silence in the low-energy stream dilutes that signal. Lei and Mirghafori find that silence alone carries speaker information (32.2% EER on ROSSI), which is evidence that background channel content is structured. Each of these papers treats non-speech as one block, either kept or dropped. None asks which properties of it generalize.
The feature-engineering direction supplies the method. Compact, domain-designed features let a simple model do what a deep model does, and they make explicit which structure is doing the work. FusionVAD is the closest audio precedent: simple addition of MFCCs to frozen pretrained embeddings beat cross-attention. In our design the engineered features are an instrument for locating and removing a shortcut, not a replacement for the detector. That is what distinguishes this proposal from the dual-band and low/high-energy systems, which engineer features to improve accuracy.
Research Program
The first stage builds the descriptor. We segment each utterance with a VAD and split the low-energy regions into boundary silence, inter-word pauses, and low-energy speech transitions. This three-way split directly addresses the dilution problem Suthokumar et al. report. From these regions we compute roughly five families of measurements. The duration and placement family covers leading and trailing silence length and the fraction of the file that is non-speech. The level family covers noise-floor energy and its variance. The spectral-shape family covers linear cepstra averaged over pauses, plus the ratio of energy above and below 4 kHz, which mirrors the split in Zhang et al. The amplitude-distribution family covers the probability mass at and near zero and the kurtosis of samples. The temporal family covers decay slope after speech offsets, as a reverberation proxy, and noise stationarity. The whole descriptor is a few dozen numbers, and every one of them has a physical interpretation.
The second stage is the audit. We fit a logistic regression on the descriptor alone for each dataset. This gives a "shortcut ceiling": how well a detector could do from non-speech properties with no speech content at all. We then run counterfactual edits on a trained AASIST model (Jung et al.), one family at a time. For example, we re-time boundary silence to match the genuine-class distribution. Separately, we transplant noise floors drawn from a genuine pool into a spoof file's pauses. We also apply a non-speech-only amplitude remap in the style of Lapidot and Bonastre. For each edit we measure the score shift and the change in EER. A family counts as a shortcut if two things hold together. First, edits to it move scores strongly. Second, its predictive power in the logistic probe drops sharply between the training corpus and a held-out corpus, such as ASVspoof 2019 LA to the 2021 DF hidden subsets. A family counts as physics if it stays predictive across corpora. Replay-relevant spectral shape and decay on PA are the families we expect to fall here.
The third stage is the intervention. During training we randomize only the families labeled as shortcuts. For example, we resample silence durations independently of class, or rescale noise-floor level. The spectral colouring and decay of the pauses stay intact. We also track Shim et al.'s ratio metric throughout, to test whether removing silence shortcuts moves genuine-class fragility at all. Their result predicts some residual bias inside speech frames, and we will report it rather than claim to remove it.
Baselines
The naive reference point is AASIST trained and tested on full, untrimmed utterances with its published recipe. It reports 0.83% EER and 0.0275 min t-DCF on ASVspoof 2019 LA (Jung et al.). This is the strongest single-model result in the file, and it is the model most likely to exploit silence, which makes it both the accuracy bar and the object of the audit. We retrain it ourselves with our seeds so that every variant shares data, epochs, and hyperparameters.
The second baseline is the same AASIST trained and tested on VAD-trimmed audio. This is the blanket-removal fix. Prior work predicts it loses in-domain accuracy, and it is the natural competitor on silence-trimmed hidden subsets.
The third baseline is the strongest existing silence-robust design. The first candidate is the dual-band SENet of Zhang et al., which reports 1.14% EER for the low-band model and 3.50% EER after VAD on ASVspoof 2019 LA. The second is the low/high-energy fusion of Suthokumar et al., which reports 0.40% average EER on SAS. We reimplement the dual-band system on our splits. The fourth baseline is AASIST trained with Shim et al.'s Generalized Cross Entropy loss, which they report moves LA 2019 EER from 1.39% to 1.35%. This is the loss-based fix.
For context on out-of-domain difficulty, the ASVspoof 2021 overview reports a best DF EER of 15.64%, a best LA min t-DCF of about 0.21, and a best PA min t-DCF of about 0.68. The file contains no reported result on Lacuna for any system evaluated specifically on the 2021 silence-trimmed hidden subsets, so we will produce those numbers for every baseline ourselves.
Two ablations isolate the key ingredient. The first randomizes an equal number of families chosen at random instead of by the audit. The second randomizes all families (blanket silence augmentation). All variants receive the same augmentation probability and training budget. Beating the baselines means two things together. The targeted model must match or beat the trimmed and blanket-randomized models on the LA and DF hidden subsets. It must also beat them on PA real-room data, while losing clearly less in-domain accuracy than trimming does.
Why not just…?
Why not just trim silence at train and test time? This is the simplest cure for the shortcut, and the 2021 hidden subsets reward it. However, four independent studies (Font et al., STC, Wang et al., Zhang et al.) report that trimming loses discriminative information, and Shim et al. show that it does not remove genuine-class bias. Our experiment settles the question directly. If trimming ties the targeted model on PA real rooms as well as on the hidden subsets, the "keep the physics" part of our claim is wrong.
Why not just randomize everything in the silence? This is the real threat. Splicing random durations and random noise floors from a corpus such as MUSAN is easy and may generalize well. Its weakness in principle is that replacing noise floors also erases the loudspeaker and room colouring that replay detection relies on. The blanket-randomization ablation is exactly this alternative under an identical budget. If it matches targeted randomization on PA, we will conclude the audit adds interpretability but not accuracy, and we will report that outcome.
Why not just change the loss, as Shim et al. do? Their GCE gain on LA 2019 is 0.04 points of EER. The paper itself says the bias survives, and a loss change gives no account of which cue is being exploited. We include GCE as a baseline and also combine it with targeted randomization, to test whether the two are complementary.
Why not just let the network learn from raw non-speech frames, with no engineered features? A network trained on pauses alone would measure how much information the pauses contain. It would not say which properties generalize, and a network's reliance on individual properties cannot be edited family by family. The engineered descriptor is what makes the counterfactual audit possible. Its logistic probe also serves as a cheap check: if it nearly reaches AASIST's in-domain accuracy, that alone indicates how much of the benchmark is solvable without speech.
Proposed Experiments
We train on ASVspoof 2019 LA and PA. We test in-domain, on the ASVspoof 2021 LA, DF, and PA evaluation sets including their hidden silence-trimmed subsets, and on SAS for a corpus with pre-trimmed silence. Metrics are EER, min t-DCF, and Shim et al.'s ratio metric under asymmetric MP3 compression and additive noise.
The first decisive check is the audit's predictive validity. Families labeled as shortcuts on LA 2019 should be the ones whose probe accuracy falls between LA 2019 and DF 2021. Families labeled as physics should keep their accuracy on PA. If the labels do not predict transfer, the decomposition is not meaningful and the intervention stage stops.
The second check compares targeted randomization against trimming and blanket randomization in the 2021 hidden-subset and real-room conditions. The claim fails if blanket randomization matches it everywhere.
The third check asks whether removing silence shortcuts lowers the genuine-class fragility ratio at all. It tells us how much of Shim et al.'s bias lives in silence and how much in speech frames.
All results are reported over three seeds with confidence intervals, since differences of a few tenths of a point of EER are within the variance seen across the cited papers.
References
- Experimental Analysis of Features for Replay Attack Detection — Results on the ASVspoof 2017 Challenge
- STC Antispoofing Systems for the ASVspoof2019 Challenge
- Robust Audio Anti-spoofing Countermeasure with Joint Training of Front-end and Back-end Models
- The Effect of Silence and Dual-Band Fusion in Anti-Spoofing System
- ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
- Beyond Silence: Bias Analysis through Loss and Asymmetric Approach in Audio Anti-Spoofing
- Effects of Waveform PMF on Anti-Spoofing Detection
- Independent Modelling of High and Low Energy Speech Frames for Spoofing Detection
- Data selection with kurtosis and nasality features for speaker recognition
- Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion
- AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks