Voice anti-spoofing detectors decide whether a speech clip came from a person or from a speech synthesizer. On the field's main benchmark, ASVspoof 2019 Logical Access (LA), they report error rates of a percent or two. Outside that benchmark, the same detectors often fail badly. A common explanation is that the models overfit. This proposal tests a second, untested explanation: the published numbers are themselves selected. Each paper reports its best seed, its best checkpoint, and its best configuration, and compares against a lightly tuned baseline. The runs that went worse stay in the "file drawer," the unreported experiments behind every result. We cannot recover other labs' file drawers. We can, however, build a complete one ourselves. We would run a large, pre-registered grid of detectors on ASVspoof 2019 LA and record every run, including failed seeds, discarded checkpoints, and baselines tuned with the same budget as the new methods. We would then replay the publication process on that record to measure how much of the apparent progress comes from selection rather than from better methods.
The deliverable is a calibrated correction that estimates the expected inflation of a single reported equal error rate (EER) given the search that produced it. EER is the operating point where false accepts equal false rejects. We would validate this correction against an outside reference: the detectors' ranking on data with different attacks and recording conditions. If a selection-corrected ranking of methods predicts out-of-distribution ranking better than the raw "best run" ranking, the community gets a concrete, checkable reason to change how results on this benchmark are reported.
Core Claim
ASVspoof 2019 LA has a structural feature that makes it vulnerable to selection bias. Its evaluation set contains 13 attack systems (A07–A19) that never appear in training (The ASVspoof 2019 database). Many papers also use this same evaluation set, in practice, to pick checkpoints and configurations. Training on this kind of data is known to be unstable: test error swings while training loss falls smoothly (Attack Agnostic Dataset). Instability combined with test-set selection is exactly what produces a winner's curse (Winner's Curse? On Pace, Progress, and Empirical Rigor).
We define the selection inflation of a reporting protocol that keeps the best of $k$ runs as
$$\Delta_k = \mathbb{E}[\mathrm{EER}] - \mathbb{E}\left[\min_{i \le k} \mathrm{EER}_i\right],$$
where $\mathrm{EER}_i$ is the evaluation EER of run $i$ and the expectation is over seeds and configurations within a method's search space. This is the gap between what a typical run achieves and what the reported run achieves.
Our claim is twofold. First, on ASVspoof 2019 LA, $\Delta_k$ for typical $k$ is comparable in size to the reported gaps between successive single-system methods. Second, once methods are compared at equal tuning budget and corrected for $\Delta_k$, a substantial part of the published ordering of architectures no longer predicts which architectures generalize. Either half could come out negative, and a negative answer is itself publishable. It would be an "existing method negative result" in the sense of Position: Embracing Negative Results in Machine Learning.
Relation to Prior Work
Three anti-spoofing papers motivate this work by showing that LA numbers do not transfer.
How to Construct Perfect and Worse-than-Coin-Flip Spoofing Countermeasures injects MP3 compression, noise, and loudness changes that correlate with the class label. Detectors then reach near-0% EER when the correlation holds between train and test, and more than 50% EER when it is flipped. That paper studies shortcuts induced by the experimenter. We study selection performed by the reporting process, and we reuse its interventions only as a stress test.
Data Quality as Predictor of Voice Anti-Spoofing Generalization shows that acoustic mismatch between corpora, such as long-term spectra and signal-to-noise ratio, explains up to 68% of EER variation for the LFCC-GMM detector. It explains where errors come from. It does not address which reported runs are representative.
Attack Agnostic Dataset documents instability across folds and builds a harder benchmark. It reports variance but never asks how that variance, filtered through publication, distorts the LA literature.
On the meta-science side, Unraveling overoptimism and publication bias in ML-driven science supplies the closest method. It fits a learning curve, an overfitting term, and a truncated-publication model to published accuracies, and it identifies bias through how reported performance varies with sample size. That identification strategy breaks on ASVspoof 2019 LA, because every paper uses the same fixed evaluation set, so sample size barely varies. This constrains our design. We therefore do not attempt a literature meta-analysis as the primary instrument. Instead we generate ground truth directly, and we use the truncated-regression model only as a competitor whose recovery we can check against our full record.
Two further papers supply the design principles: the winner's-curse paper (equal tuning budgets, sliced analysis) and the negative-results position paper (pre-registration and post-mortem analysis). Neither audits a speech benchmark.
Research Program
We would pre-register a grid covering six detector families that span the literature:
- LFCC-GMM and CQCC-GMM, the classical generative baselines that, according to Hanilçi et al., may generalize better than discriminative classifiers;
- an LFCC light CNN (LCNN);
- RawNet2;
- a raw-waveform ResNet in the style of RW-Resnet;
- a reimplementation of the Feature Pyramid Conformer with elastic-margin softmax (Synthetic Voice Spoofing Detection based on Feature Pyramid Conformer).
Each family gets the same hyperparameter-search budgets, at three levels (1, 10, and 50 configurations), and each configuration is trained with 10 seeds. Every checkpoint is scored. For each checkpoint we log overall EER, minimum tandem detection cost function (min t-DCF), per-attack EER, and the full score distributions. The t-DCF is the challenge's cost measure that pairs the detector with a fixed speaker-verification system. All runs are released as an open record, in the spirit of ARA: Agent-Native Research Artifacts.
On this record we replay several reporting protocols:
- picking the checkpoint on the development set versus on the evaluation set;
- reporting the best seed versus the mean;
- tuning the proposed method at the high budget while tuning the baseline at the low one.
For each protocol we estimate $\Delta_k$ and the probability that a claimed improvement of a given size between two families survives under a matched protocol. From these estimates we fit a correction. Given a reported EER, a stated or inferred search size, and the method family's run-to-run spread, the correction returns an estimate of typical-run performance and an interval around it. Finally, we sort failures by attack system to identify which of A07–A19 stay hard for every family no matter how much tuning is done. That gives the field-level negative result that Shim et al. explicitly left open.
Baselines
The naive reference point is face value: methods are ranked by their single best published-style number. In our grid, that means the best evaluation-selected run per family. Any correction must beat this ranking to be worth using.
The strongest existing reporting practice is mean and standard deviation across seeds, which the Attack Agnostic Dataset paper uses. We compute it from exactly the same runs. Mean ± std captures seed noise. It does not capture configuration search or checkpoint selection on the evaluation set, so it is the natural competitor for our correction.
The strongest existing statistical de-biasing method is the truncated-regression observation model of Unraveling overoptimism. We adapt it to EER by treating lower values as better and the truncation as a cap on reported error. We fit it to "published" numbers sampled from our record through each simulated protocol, and we score it on whether it recovers the known typical-run EER.
For reference performance on the benchmark itself:
- The best single system in the file is FP-Conformer, at EER 1.65% on the LA evaluation set without data augmentation (Synthetic Voice Spoofing Detection based on Feature Pyramid Conformer).
- The top challenge fusion reported EER 0.22%, with min t-DCF given as 0.0069 in Future Horizons and as 0.0692 in the challenge overview. We would note this discrepancy rather than resolve it.
- No reported result was found on Lacuna for the LFCC-GMM or CQCC-GMM evaluation EER, so we would report our own from-scratch numbers.
Our reimplemented FP-Conformer should land near 1.65% in its best run. If it does not, that is itself a replication finding. We would not count it as evidence of selection, because an implementation gap and a selection effect cannot be told apart from that number alone.
The key ablation equalizes tuning budget while leaving checkpoint selection unchanged, and then does the reverse. This separates baseline under-tuning from evaluation-set selection.
"Beating" the competitors means two things. First, the corrected ranking of the six families must correlate better, measured by Kendall's rank correlation, with their ranking on out-of-distribution data than both the face-value ranking and the mean ± std ranking do. Second, the correction must recover typical-run EER with lower error than the truncated-regression model.
Why not just…?
Why not just report mean and standard deviation over seeds? This is the real threat. If seed variance accounts for most of $\Delta_k$, the recommendation reduces to "report seed statistics," which is cheap and already advocated. Our grid settles the question directly: we decompose the total inflation into a seed share, a checkpoint-selection share, and a configuration-search share. If the seed share dominates, we say so and recommend the simpler practice. We expect checkpoint selection on the evaluation set to matter on its own because training is unstable (Attack Agnostic Dataset), but that expectation is what the experiment tests.
Why not just meta-analyze the published ASVspoof 2019 LA literature with truncated regression? That model identifies bias through variation in sample size (Unraveling overoptimism). Here every paper shares one fixed evaluation set, and the field has no ground truth against which to check the estimate. Our record supplies that ground truth, so we can measure how far a literature-only fit misses before anyone trusts one.
Why not just tune the baselines properly, as the winner's-curse paper recommends? We do. It appears as a condition in the grid, and the budget-matching ablation measures its effect alone. Equal tuning removes one source of bias. It does not tell a reader how to discount a number that was already published under unequal tuning, and producing that discount is the purpose of the correction.
Why not just move to a harder benchmark with disjoint attacks? A new benchmark resets the leaderboard without changing how results are selected, so the same inflation would reappear on it (Position: Embracing Negative Results). We use the harder benchmark as the outside reference, not as a fix.
Proposed Experiments
All training uses the ASVspoof 2019 LA train partition, and the development and evaluation partitions are used as specified by each simulated protocol. Out-of-distribution reference performance comes from two sources:
- the Attack Agnostic Dataset folds (ASVspoof 2021 LA, WaveFake, FakeAVCeleb audio), evaluated without retraining;
- the shortcut interventions of Shim et al. (MP3 compression and additive noise) applied to the LA evaluation audio.
We report metrics at three levels. The primary metrics are EER and min t-DCF. We also report per-attack EER and Kendall's rank correlation between families' LA rankings and their out-of-distribution rankings.
The first decisive check is whether $\Delta_k$ under evaluation-set selection at $k = 10$ is at least as large as the 1.65%-to-baseline-scale gaps that separate the single-system families. If $\Delta_k$ is small, the selection hypothesis fails, and we report that LA progress is real at the family level.
The second check is whether the corrected ranking predicts the out-of-distribution ranking better than the face-value ranking. We pre-register the comparison and test it with a bootstrap over seeds.
The third check is whether any attack system stays above a pre-registered EER threshold for all six families at the highest tuning budget. Such an attack would be a field-wide boundary condition rather than a weakness of any one method.
A null result on all three checks would still be worth publishing. It would establish, with a full record, that the benchmark's headline numbers are honest representatives of their methods.
References
- The ASVspoof 2019 database
- Attack Agnostic Dataset: Towards Generalization and Stabilization of Audio DeepFake Detection
- Winner's Curse? On Pace, Progress, and Empirical Rigor
- Position: Embracing Negative Results in Machine Learning
- How to Construct Perfect and Worse-than-Coin-Flip Spoofing Countermeasures: A Word of Warning on Shortcut Learning
- Data Quality as Predictor of Voice Anti-Spoofing Generalization
- Unraveling overoptimism and publication bias in ML-driven science
- Classifiers for synthetic speech detection: a comparison
- RW-Resnet: A Novel Speech Anti-Spoofing Model Using Raw Waveform
- Synthetic Voice Spoofing Detection based on Feature Pyramid Conformer
- ARA: Agent-Native Research Artifacts
- ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
- ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech