Singing voice synthesis turns a musical score (lyrics, notes, durations) into a sung waveform. The strongest systems are diffusion models, which generate by gradually denoising random noise into a spectrogram. They sound good, but their internal architectures are still chosen by hand. The usual fix, automated architecture search, is expensive. Its cheap shortcuts, such as scoring untrained networks or training candidates briefly, were validated on image classification. For image generators they are known to rank architectures backwards. We think singing has a sharper version of this problem. An architecture can look fine on average spectral error and still fail on the rare high notes, vibrato, and glides that listeners notice. A cheap ranking signal that tracks the average may therefore pick exactly the wrong model.
This project has two parts. First, we build a small benchmark of singing-synthesis denoisers, each trained to convergence and scored on aggregate quality, on rare-pitch and technique-specific accuracy, and on speed. With it we measure where cheap ranking methods break. Second, we test one specific cheap estimator. It trains each candidate briefly and then scores it only at the noise levels the model actually uses at inference, and only on score segments weighted toward rare pitches and singing techniques. The two-sentence version is this: cheap architecture search for singing should be judged by whether it ranks models correctly on the hard tail of the music, not on average loss. We provide the ground truth to check that, and we test a tail-matched short-training estimator against it.
Core Claim
Three findings motivate the claim. Auto-GAS showed that classification-derived zero-cost proxies can anti-correlate with generative quality. On TransNAS-Bench-101, Synflow reached Kendall's τ of −50.91% and Zen-NAS −2.27%, while Auto-GAS's evolved proxy reached 61.14%. That work covered only image GANs and explicitly left diffusion and audio open. EZNAS reports that zero-cost proxies lose discrimination among the top 10% of architectures, which is the only region a search cares about. Opencpop documents that singing models degrade on long-tail pitches and extreme registers.
Putting these together, we claim the following. The rank agreement between cheap estimators and fully trained singing models will be much lower on tail metrics than on aggregate metrics. It will be lowest in the top decile. An estimator that measures the inference-relevant part of the denoising task on tail-weighted data will recover much of that lost agreement at the same compute.
No prior work in the file runs architecture search over singing synthesis models. The closest audio search work, Conformer Space NAS for audio separation, searches discriminative separators with conventional training-based search. It uses no cheap estimation and no generative objective.
Research Program
Search space. The search space is built around DiffSinger. DiffSinger uses a fast auxiliary decoder to produce a blurry spectrogram. It then adds noise up to an intermediate step $k$ and runs a WaveNet-style denoiser only from $k$ back to clean. That design cut its real-time factor (compute time per second of audio) from 0.348 to 0.191 on PopCS.
We keep the auxiliary decoder, the note and duration front-end, and the vocoder fixed. We search only the denoiser, along the following dimensions:
- depth and width;
- dilation schedule;
- a choice per block between isotropic 1D convolution, rectangular 2D kernels of varying time-to-frequency aspect ratio, and the horizontal/vertical stripe pooling that SUSing hand-designed (in SUSing this lowered mel-cepstral distortion from 9.36 to 8.42 on Kiritan);
- the shallow-diffusion boundary $k$ as a searched variable, since architecture capacity and the number of denoising steps trade off against each other for latency.
SUSing supplies the operator family. DiffSinger supplies the backbone. Neither searched their own configuration.
Ground-truth benchmark. We train roughly 120 sampled architectures to convergence on Opencpop, using its official split. Opencpop has 5.2 hours of audio with manual phoneme, note, and slur labels. We budget about half a GPU-day per model, so the benchmark costs about 60 GPU-days. This cost is the "efficiency paradox" that the NAS literature warns about. We pay it once and release the trained models and scores, so later estimators can be evaluated for free.
Each model is scored on:
- mel-cepstral distortion;
- F0-RMSE (pitch error);
- F0-RMSE restricted to the rarest 10% of note bins;
- slur-segment error;
- measured real-time factor.
A 40-architecture subset is also trained on the Mandarin and English portions of GTSinger. Its phoneme-level vibrato, glissando, falsetto, and breathy labels allow per-technique pitch and spectral error. This subset tests whether rankings transfer across corpora.
The estimator. Let $a$ be an architecture and $\theta_a^{(S)}$ its weights after $S$ short training steps. Let $D_j$ be held-out segments in stratum $j$, where strata are defined by pitch-frequency bin and technique label. Let $w_j$ be stratum weights inversely proportional to training frequency. Let $x_t$ be a spectrogram noised to step $t$, $\epsilon$ the added noise, $\epsilon_\theta$ the model's noise prediction, and $c$ the score conditioning. The score is
$$\hat{s}(a)=\sum_j w_j\,\mathbb{E}_{t\le k,\;x\in D_j}\big\|\epsilon-\epsilon_{\theta_a^{(S)}}(x_t,t,c)\big\|^2 .$$
The restriction $t\le k$ comes from shallow diffusion: the network is never asked to denoise above step $k$. Loss at high noise levels spends estimator signal on a task the deployed model never performs. The stratum weighting targets the tail metric directly.
We choose $S$ and data fraction using the budget-selection procedure of Rethinking Performance Estimation in NAS, which uses random-forest sensitivity pruning of estimation hyperparameters. That paper found that epochs and depth matter for rank preservation while input resolution can be cut. We test whether the same holds when "resolution" means audio sampling rate and segment length. We then run an evolutionary search that uses $\hat{s}$ as fitness and measured real-time factor as a second objective.
Baselines
Default reference. The do-nothing reference is DiffSinger's hand-set denoiser, trained on Opencpop with our fixed front-end and vocoder. We found no reported result on Lacuna for DiffSinger on Opencpop. The nearest reported figures are DiffSinger's PopCS MOS of 3.85 at real-time factor 0.191, and the Opencpop CpopSing baseline (a Conformer FastSpeech 2) at F0-RMSE 27.58 and MOS 3.70. We retrain both CpopSing and default DiffSinger ourselves, so every number shares data, vocoder, and listening panel. A random-search baseline with the same GPU-hour budget as our search sets the floor that any estimator must beat.
Strongest existing ranking method. This is an Auto-GAS-style evolved proxy. We run Auto-GAS's genetic-programming proxy discovery and information-bottleneck primitives on the training half of our benchmark architectures, then evaluate it on the held-out half. Its only reported figure is Kendall's τ of 61.14% (Spearman 0.79) against SSIM on TransNAS-Bench-101. We report no audio number for it until we measure one. We also include:
- the classic zero-cost proxies Synflow and NASWOT;
- the training-free trainability, expressivity, and generalization indicators of TEG-NAS;
- plain truncated training scored by ordinary validation loss, at budgets chosen by the BPE procedure.
Every estimator gets the same GPU-hours per candidate and the same held-out architectures. Evolved proxies get the same number of architecture labels for fitting as $\hat{s}$ gets for choosing its weights.
Ablations. Two ablations isolate the key ingredients. One drops the $t\le k$ restriction, using full-range denoising loss. The other drops stratum weighting, using the uniform held-out average. Beating the baselines means two things. First, a statistically higher top-decile Kendall's τ on rare-pitch F0-RMSE than both BPE-truncated loss and the evolved proxy, with bootstrap intervals over architectures. Second, a searched architecture that beats default DiffSinger in a blind MOS and preference test at equal or lower measured real-time factor.
Why not just…?
Why not just keep DiffSinger's architecture and tune by hand? For average quality this may well be enough, and the default-model baseline is included to find out. The case for search rests on the tail. SUSing shows that operator geometry alone moves spectral error substantially. No hand-design study in the file reports rare-pitch or per-technique error across architectures. If the benchmark shows the default already sits on the quality–speed frontier for tail metrics, that negative result is reported as the outcome.
Why not just train each candidate briefly and rank by validation loss? This is the real threat. BPE made this approach 1,000× cheaper than AmoebaNet on images, and plain short training may already rank singing denoisers well. We treat it as the main competitor, not a strawman. It gets the same budget-selection procedure as our estimator. The two ablations test directly whether noise-level restriction and tail weighting add anything beyond it. If they do not, the benchmark still establishes that simple truncated training is sufficient for singing. That would be a useful, citable finding, and we would recommend it over our estimator.
Why not just use an existing zero-cost proxy, or Auto-GAS's? Auto-GAS's own results show classification proxies failing on generation. Its evolved proxy is untested on diffusion or audio, and EZNAS reports that proxies weaken in the top decile. We do not assume the proxy fails. We run it on the benchmark under identical conditions.
Why not a weight-sharing supernet? Rethinking Performance Estimation and TEG-NAS both report that supernet rankings diverge from standalone trained networks and collapse toward parameter-free operations. As a cheap check, we train one supernet over the search space and report its rank agreement on the benchmark alongside the other estimators.
Decisive Checks
The first check tests the premise. If every cheap estimator shows top-decile rank agreement on rare-pitch and technique error that is about as high as on aggregate mel-cepstral distortion, the tail-mismatch hypothesis is false. The project then reduces to a benchmark release.
The second check tests the estimator. At matched GPU-hours, $\hat{s}$ must beat BPE-truncated validation loss on tail rank agreement on Opencpop. It must also keep that advantage on the GTSinger transfer subset, where technique labels allow vibrato and glissando to be scored separately.
The third check tests practical value. The architecture chosen by the tail-aware search must win a blind listening test against default DiffSinger and CpopSing on rare-note and technique-heavy excerpts, at no higher measured real-time factor. Total search cost, excluding the one-time benchmark, must stay under a few GPU-days, so that the search does not cost more than the models it compares.
References
- Auto-GAS: Automated Proxy Discovery for Training-Free Generative Architecture Search
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis
- Conformer Space Neural Architecture Search for Multi-Task Audio Separation
- DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
- SUSing: SU-net for Singing Voice Synthesis
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks
- Rethinking Performance Estimation in Neural Architecture Search
- Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics