A voice activity detector is the small, always-running model that decides, frame by frame, whether someone is speaking. On earbuds, hearing aids and phones it has to fit in a few thousand parameters and run in well under a millisecond per frame. Designing such micro-models automatically, with neural architecture search (an algorithm that proposes candidate network designs and keeps the best), is attractive. It is also expensive, because each candidate must be trained to be judged. The literature offers many ways to judge candidates cheaply: score the untrained network with one gradient pass, train it for a few epochs, or fit a predictor on such scores. Every one of these tricks was validated by asking whether the cheap score ranks architectures the same way as full training on a matched test set, and almost always on vision.
Voice activity detection breaks that assumption in a specific way. The real test is noisy, real-world audio (movie soundtracks with music and effects) that looks very different from the synthetic noise mixtures used for training and validation. Our claim is that for tiny detectors, the choice of audio the cheap score is computed on determines whether the ranking transfers to real noise far more than how few epochs or FLOPs the score uses. We build a fully trained benchmark of sub-20k-parameter detectors to test this. We then use a condition-stratified short-training proxy to search for a detector that beats a hand-scaled mobile baseline at equal parameter count.
Core Claim
The only published search for voice activity detectors, NAS-VAD, runs Bayesian optimization over attention-rich spectrogram cells. It fully trains each candidate and finds a 151k-parameter model reaching 0.905 AUC on AVA-Speech. That model is an order of magnitude larger than the hand-designed ResectNet, which works on raw audio with learnable sinc filters, grouped convolutions, a channel-shifting module and a GRU. ResectNet reaches 90.0% AUC-ROC on AVA-Speech with 11.1k parameters, and loses only 1.4 points at 4.5k. Neither paper asks the question that matters for cheap search in this regime: does a cheap score's ranking of micro-architectures survive the move from synthetic training mixtures to real noise?
There is reason to fear it does not. NAS-Bench-ASR found that among top speech-recognition models, validation and test rankings decouple much more than in vision. The AVA-Speech paper shows that detectors degrade sharply under its "Speech + Noise" and "Speech + Music" conditions. Our hypothesis is concrete: a short-training proxy validated on synthetic mixtures stratified to match those hard conditions (low signal-to-noise, music-overlapped speech) will rank micro-architectures by real-world AUC substantially better than the same proxy validated on clean or uniformly mixed audio. It will also beat zero-cost proxies once model size is controlled for. If that holds, search for tiny detectors becomes cheap and reliable. If it fails, the benchmark itself documents where cheap ranking breaks for audio.
Research Program
The first component is a tabular benchmark of tiny detectors, in the spirit of NAS-Bench-ASR. The search space generalizes ResectNet's macro-structure. At each of four stages it chooses the front end (learnable sinc filters on raw audio versus log-mel features), the temporal kernel size, the pointwise group count, and whether to include the channel-shifting module. It also chooses the sequence layer (GRU, small time-axis attention, or none). We borrow from NAS-VAD the option of reductions that halve frequency while keeping full time resolution. Every model is constrained below 20k parameters. We sample roughly 1,500 architectures, stratified across parameter bins (under 5k, 5–10k, 10–20k).
Each architecture is trained with three seeds on the Interspeech 2020 DNS-Challenge mixtures, the same training data ResectNet used. We checkpoint at several budgets so that short-training proxies can be read off the table. Every model is evaluated on AVA-Speech overall and per condition, and on ACAM and HAVIC as secondary real-world sets. We record wall-clock time, including data loading and feature extraction, and CPU latency. At an estimated one GPU-hour per full run, this is several thousand GPU-hours, spent once. Releasing the table lets others test proxies for free, as NAS-Bench-Suite-Zero did for vision.
The second component is the analysis. On this table we compute every cheap score's rank agreement with fully trained AVA-Speech AUC. We report Kendall's tau overall, within each parameter bin (to strip out the size bias that NAS-Bench-Suite-Zero documents), and within the top decile, where EZNAS reports that zero-cost proxies fail. The key manipulation holds the training budget fixed and varies only the validation audio the proxy is computed on. The variants are clean speech, uniform synthetic mixtures, and a condition-stratified set whose signal-to-noise and music-overlap distribution is matched to AVA-Speech's published condition proportions. The matching uses only those published proportions, never AVA-Speech audio or labels.
The third component is the search itself. We run evolution under a fixed wall-clock budget, scoring candidates with the best-ranking proxy from the analysis. Size-residualized zero-cost scores serve as a pre-filter only if the analysis supports it. The winners are then fully trained and compared to hand-scaled ResectNet at matched parameter counts.
Baselines
The naive reference point is ranking candidates by parameter count or FLOPs. NAS-Bench-Suite-Zero found that FLOPs alone is a surprisingly stable proxy across tasks, so any cheap score must beat it within a parameter bin to show it measures anything beyond size. We also include random search under the same wall-clock budget.
The strongest competitor on the final detector is ResectNet scaled by width multiplier. It reports 90.0% AUC-ROC on AVA-Speech at 11.1k parameters and 88.6% at 4.5k (ResectNet). It is the strongest because it already sits in the target regime and needs no search at all. We retrain it ourselves on the same DNS-Challenge pipeline, with the same augmentation, seeds and tuning budget as our searched models. Beating it means higher mean AVA-Speech AUC at equal or smaller parameter count and equal or lower CPU latency, with the gap exceeding seed-to-seed variation, at both the roughly 4.5k and roughly 11k points.
The strongest existing search method is NAS-VAD-style Bayesian optimization with full training, which reports 0.905 AUC on AVA-Speech at 151k parameters (NAS-VAD). We rerun it inside our sub-20k space with the same wall-clock budget as our proxy-driven search. Since the table holds full-training results, it can also be simulated exactly.
The strongest existing cheap-scoring methods are three. The first is the Synflow, jacob_cov and SNIP voting ensemble from Zero-Cost Proxies for Lightweight NAS, which reports Spearman 0.82 on NAS-Bench-201. The second is the evolved EZNAS-A proxy, with Kendall's tau 0.65 on NAS-Bench-201 CIFAR-10 (EZNAS). The third is an XGBoost surrogate fed all standard zero-cost proxies, following NAS-Bench-Suite-Zero. The surrogate is trained on a held-out portion of our table so that it gets the same labeled-data budget our proxy selection uses. No reported result found on Lacuna for any of these on voice activity detection. Budgeted few-epoch training in the style of Rethinking Performance Estimation is included at matched cost.
The ablation that isolates our key ingredient compares identical short-training proxies that differ only in validation audio: clean versus uniform-mixture versus condition-stratified.
Why not just…?
Why not just scale ResectNet's width and skip search? This is a real threat. ResectNet loses only 1.4 AUC points when halved, which suggests its operation mix is already good. What favors search is only NAS-VAD's finding that searched attention cells beat a hand-designed spectro-temporal model while being 3.7x smaller, and that finding sits at a different scale. The matched-parameter comparison settles the question directly. If no searched model beats width-scaled ResectNet beyond seed noise, we report that search does not pay below 20k parameters. The benchmark and ranking analysis remain useful either way.
Why not just train a big detector and quantize it? The noisy-conditions paper (Optimizing Voice Activity Detection for Noisy Conditions) cut Snapdragon CPU load from 37% to 14% with quantization and a vendor runtime. However, that model is a 2D-CNN with a denoising autoencoder that is far larger than the sub-20k budget, and it reports accuracy on AISHELL-1 mixtures rather than AVA-Speech AUC. Quantization is also orthogonal to our question: searched and hand-designed micro-models can both be quantized. We will quantize the final models of every method identically, so this alternative becomes a shared post-processing step rather than a competitor.
Why not just use a few epochs of training, which already transfers well in speech? NAS-Bench-ASR reports roughly 0.85 ranking correlation between TIMIT and LibriSpeech. That is clean-to-clean transfer, and it is exactly the kind of agreement we expect to collapse under real noise. For tiny models, few-epoch training is also genuinely cheap. This makes it the proxy we build on rather than one we try to replace, and our contribution is choosing what it is validated on. The ablation shows directly whether a plain few-epoch proxy already matches the stratified one. If it does, the claim fails and we say so.
Why not just run Synflow or jacob_cov on spectrograms? Zero-Cost Proxies for Lightweight NAS found Synflow the most robust standalone proxy, including on speech recognition. But NAS-Bench-Suite-Zero shows that these scores largely track model size, and our space mixes sinc filters, GRUs and attention, topologies that EZNAS admits its proxies do not handle. The within-bin and top-decile correlations will show whether they rank anything beyond parameter count.
Proposed Experiments
The first decisive check is the stratification ablation. The claim holds if, at a fixed short-training budget, the condition-stratified proxy's within-bin Kendall's tau against AVA-Speech AUC exceeds both the clean-validated and uniform-mixture versions, by a margin larger than the seed-level bootstrap interval. The gain should be concentrated in the "Speech + Noise" and "Speech + Music" condition rankings.
The second check asks whether any zero-cost score, or the surrogate built on all of them, beats parameter count and FLOPs within bins and in the top decile. We report wall-clock cost per candidate for each score, including data loading, so that the speedup claims face the gap between theoretical and wall-clock savings that Zero-Cost Proxies for Lightweight NAS observed.
The third check is the end-to-end result under a fixed wall-clock budget. We compare the best model found by stratified-proxy evolution, Bayesian optimization with full training, random search, and zero-cost-guided search against retrained width-scaled ResectNet. The comparison is on AVA-Speech AUC, ACAM and HAVIC, at around 4.5k and 11k parameters, with CPU latency reported.
AVA-Speech is never used for proxy design, tuning, or selection. Its segment-level labels, which ResectNet and the AVA-Speech authors note are coarse, are handled with the standard frame-level scoring those papers use.
References
- NAS-VAD: Neural Architecture Search for Voice Activity Detection
- ResectNet: An Efficient Architecture for Voice Activity Detection on Mobile Devices
- NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition
- AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Zero-Cost Proxies for Lightweight NAS
- Rethinking Performance Estimation in Neural Architecture Search
- Optimizing Voice Activity Detection for Noisy Conditions