Reading Curvature to Stop Greedy Hyperparameter Search From Breaking Small Speech Models

Small, fast speech synthesizers are hard to train. Examples include the 1.46M-parameter HiFi-GAN vocoder (the network that turns a spectrogram into a waveform) and end-to-end text-to-waveform models like JETS. Their losses mix reconstruction, duration and adversarial terms, and the right balance of learning rates and loss weights decides whether training ends in clean audio or in a buzzy, artifact-ridden solution. Online hyperparameter search could find good settings within roughly the cost of a few training runs. The best-known example is population-based training (PBT), which trains several copies in parallel and periodically copies the weights of the best-scoring ones onto the worst while perturbing their settings. PBT has a known flaw, though. It picks winners by their score right now, and settings that look best in the short run often lead to worse final models.

We propose to give the selection step a second piece of evidence that is nearly free: how curved the loss surface is around each candidate. The estimate comes from the difference between consecutive gradients, which is the same signal the AGD optimizer uses to precondition its updates. Our bet is that a vocoder settling into a sharp, badly conditioned basin can be recognized by its curvature before its validation score shows it. The proposal tests this bet directly. It first asks whether curvature predicts final audio quality better than loss history alone. Only if it does, we build that prediction into PBT's selection rule and compare against the strongest anti-greedy population method in the file.

Core Claim

During training of a lightweight vocoder or end-to-end synthesizer, per-layer summaries of a cheap diagonal curvature estimate predict the rank of a training run's final spectral distortion. They add predictive power beyond the run's current validation loss and its recent loss trend. Replacing "current score" with "predicted final score" in PBT's exploit step should then reduce greedy collapse. Because only training hyperparameters are searched and the architecture stays fixed, every gain in quality comes at an unchanged real-time factor (the ratio of compute time to audio duration).

What is new is the use of curvature as a look-ahead signal for selection between models in online hyperparameter search. Prior work uses curvature only to precondition a single model's update or to prune weights. Prior work against greedy collapse relies on population structure or novelty, never on landscape shape.

Where This Sits

Three papers motivate the proposal by documenting the failure. To Be Greedy, or Not to Be shows that PBT variants, especially the Bayesian ones, over-optimize for the next checkpoint and end worse. It also shows that the update interval acts as a dial on greediness. However, it offers no signal for telling a deceptive basin from a good one. MF-PBT fixes collapse structurally. It splits the population into fast-updating and slow-updating agents and lets slow agents take winning weights while keeping their own hyperparameters. EPBT periodically selects on behavioral novelty instead of accuracy. Both methods were evaluated only on reinforcement learning or image classification, and both remain blind to the loss surface. Our proposal is complementary: it changes what an agent is ranked by rather than how the population is organized. That means it can in principle be stacked on MF-PBT, and we test that combination.

Two papers supply the curvature method. AGD shows that differences between successive gradients give a diagonal curvature surrogate at about 7% wall-clock overhead over AdamW, against nearly 3x for AdaHessian. We reuse this estimate as a measurement rather than as a preconditioner. ARNCG and GraSP show that Hessian-vector products can be computed with linear memory. We use them only occasionally, to check that the cheap secant estimate tracks a true Hutchinson-style diagonal.

The speech papers define the regime. HiFi-GAN and JETS train multi-loss adversarial systems with fixed, hand-chosen Adam settings and loss weights. SpeedySpeech does the same for a convolutional spectrogram model. Wiemerslage et al. warn that untuned baselines distort architecture comparisons in sequence models, which suggests small TTS models may sit well below their ceiling.

One constraint cuts against us. The curvature of a generator–discriminator game is not the curvature of a single loss, and it is frequently indefinite. We measure curvature of the generator objective only, and we track the sign of the curvature as well as its magnitude.

Research Program

The curvature signal. Every $K$ steps, each agent evaluates its generator gradient twice on the same minibatch, at its current parameters and one update later. Using the same minibatch keeps minibatch noise out of the difference. The extra cost is one additional gradient evaluation per $K$ steps. Let $s$ be the parameter step and $y$ the gradient difference. The elementwise secant curvature is

$$h_i = \frac{|y_i|}{|s_i| + \epsilon},$$

where $i$ indexes parameters and $\epsilon$ is a small constant. We summarize $h$ per layer by its median, its 99th percentile, and their ratio, which serves as a conditioning proxy. We also record the per-layer fraction of $y_i s_i < 0$, an indicator of negative curvature. The features fed to the predictor are these summaries, their trend over the last few windows, the current validation loss, and the loss trend.

Phase 1: a predictive study that decides whether to proceed. We train about 64 HiFi-GAN V3 runs to completion on LJSpeech. Each run uses a randomly sampled training configuration covering generator and discriminator learning rates, Adam betas, learning-rate decay, and the weights on the mel and feature-matching losses. Curvature is logged throughout. These runs double as the first systematic hyperparameter-sensitivity map for a lightweight vocoder, measured with Wiemerslage et al.'s metrics such as best-equivalent run fraction and rank correlation.

At checkpoints at 5%, 10% and 25% of training, we fit a small ranking model to predict final mel-cepstral distortion (MCD, a standard spectral distance between synthesized and real speech). We compare three feature sets: current validation loss alone, loss history alone (a learning-curve extrapolator), and loss history plus curvature. The success criterion is a gain in Spearman rank correlation, under held-out-run cross-validation, for history-plus-curvature over history alone. We also run a permutation control that shuffles curvature features across runs. If curvature adds nothing, the proposal becomes a careful learning-curve-guided PBT study for TTS, and we say so.

We also check how well the secant estimate tracks a true Hutchinson diagonal computed from Hessian-vector products on a held-out batch.

Phase 2: curvature-informed selection. In the exploit step, PBT ranks agents by the Phase-1 predictor's estimate of final quality instead of current validation loss. The predictor is trained on LJSpeech runs and kept frozen, so we can test whether it transfers. We apply this rule to standard PBT and to MF-PBT. We then move to JETS, where joint acoustic-plus-vocoder training makes loss-weight balance most fragile. We also apply the frozen predictor to the multi-speaker VCTK corpus and to the Hi-Fi-CAPTAIN corpus to test transfer across data.

Baselines

The do-nothing reference is the published default configuration of each architecture, HiFi-GAN V3 and JETS, retrained locally with three seeds. Retraining locally keeps listening tests comparable across conditions, rather than comparing against scores from another lab. For reference, JETS reports MOS 4.02 ± 0.07 and MCD 7.16 on LJSpeech. HiFi-GAN reports MOS 4.36 for its large V1 model. We found no reported MOS on Lacuna for V3. If any search method beats the defaults, that alone measures the tuning gap in efficient TTS.

Random search at equal GPU-hours is the naive search reference. It is also the method Wiemerslage et al. found sufficient to overturn architecture rankings.

Standard PBT with random perturbation is the direct parent of our method. Following Chebykin et al., it is run at two update intervals, because a longer interval is the cheapest known fix for greediness.

The strongest existing competitor is MF-PBT. It is the only method in the file that specifically targets the hyperparameter collapse we study, and its reported result is an IQM reward of 23,793 on Brax Humanoid at 1B steps, against 16,171 for PBT and 17,713 for random search. We found no reported result on Lacuna for MF-PBT on any speech task. We will implement it with the paper's fast/slow split and asymmetric weight migration.

Every method gets the same search space, population size (8 agents for HiFi-GAN V3, 6 for JETS), total GPU-hours, data splits and evaluation. Every search is run with three seeds.

The key ablation keeps everything in Phase 2 identical but drops the curvature features from the predictor. This isolates curvature from the generic benefit of predicting the future rather than reading the present. A second ablation uses AGD as the training optimizer with default settings and no search.

Beating MF-PBT means one of two outcomes. The first is lower final MCD in at least two of three search seeds, with listening-test MOS no worse within confidence intervals. The second is matching MF-PBT's quality with half its population.

Why not just…?

Why not just use MF-PBT, or plain PBT with a longer update interval? This is the most serious threat. Both attack greediness without any new signal, and Chebykin et al. show the interval alone shifts outcomes. We therefore run both as baselines and also run our selection rule on top of MF-PBT. If curvature-informed MF-PBT does not beat plain MF-PBT at equal compute, the curvature signal is redundant with population structure, and we will report that.

Why not just extrapolate learning curves from the loss history? This may capture most of the benefit, since a run heading into a bad basin may already show a flattening loss. Phase 1 is designed to settle exactly this question. The history-only predictor is the comparison that curvature must beat, and Phase 2 carries the same ablation. We commit to dropping curvature if the history-only predictor matches it.

Why not just swap Adam for a curvature-aware optimizer like AGD and skip the search? AGD's ImageNet gain over AdamW (70.35% against 69.11% top-1) shows that better preconditioning helps a single run. But the optimizer does not choose loss weights or the generator-to-discriminator learning-rate ratio, which are the knobs that make adversarial vocoders brittle. The AGD-with-defaults ablation measures how much of the gap an optimizer change closes on its own.

Why not just use a standard warmup-plus-cosine schedule and add noise? That is what the published defaults already do, so the do-nothing baseline covers it. MF-PBT also reports that mutation alone does not prevent collapse when greedy weights spread through the population.

Decisive Checks

Three results would make or break the claim. The first is the Phase-1 rank-correlation gain of curvature over loss history at the 10% checkpoint, together with the permutation control showing the gain disappears when curvature is shuffled across runs. The second is the head-to-head against MF-PBT on HiFi-GAN V3 and JETS at matched GPU-hours. We report final MCD, log-F0 RMSE (pitch error), and a MOS listening test on the selected models. Because architectures are fixed, we also confirm that real-time factor is unchanged. The third is predictor transfer from LJSpeech to VCTK and Hi-Fi-CAPTAIN without refitting. This tells us whether curvature captures something general about these adversarial loss surfaces or merely memorizes one dataset.