Neural vocoders are networks that turn a compact acoustic description, such as a mel-spectrogram, into a waveform. They often produce muffled voices with blurred resonances. The effect is worst for high-pitched speech and singing. The usual fixes are better discriminators (networks trained to tell real audio from generated audio) or slower generators such as diffusion models. We argue that part of the problem lies elsewhere: in how training losses and evaluation metrics measure the vocal tract's resonances, the formants. A voiced spectrum is not a smooth curve. It is a comb of harmonics at multiples of the pitch $F_0$, with the formant envelope visible only at those teeth. When $F_0$ is high the teeth are sparse. Any analysis that weights every sample or frequency bin equally then latches onto individual harmonics instead of the true resonances.
Classical speech analysis solved this decades ago with weighted linear prediction. The idea is to fit an all-pole filter, a model of the vocal tract as a set of resonances, while trusting mostly the brief "closed phase" of each glottal cycle. In that interval the vocal folds are shut and the tract rings freely. We propose turning that estimator into a differentiable loss for vocoder training and into a diagnostic metric for over-smoothing, called the closed-phase envelope loss. The two-sentence version is this. Score a generated waveform's formants the way a careful phonetician would, by fitting resonances only where the glottis is closed, and train the vocoder against that score. If harmonic bias is part of the blur, the gains should grow with pitch.
Core Claim
The literature treats harmonic suppression weighting as an analysis tool, and it treats spectral over-smoothing as a generator problem. Our claim is that they meet in the objective function. The reconstruction losses used by UnivNet, APNet and SingGAN penalize all time-frequency bins uniformly. At high $F_0$, the low-energy valleys between harmonics carry little information about the envelope. Kawahara et al. formalize this as irregular frequency-domain sampling of the vocal tract response. A uniform loss therefore mixes two separate errors: getting the harmonic comb right (a source property) and getting the resonances right (a filter property). When pitch-synchronous detail is uncertain, a regression loss averages it away. Flow-Based Unconstrained Lip to Speech Generation identifies this averaging as the root of blur.
Our loss isolates the filter error. Unlike the uniform losses, it estimates that error with a method known to be unbiased by harmonics at high pitch. The specific claims are three. First, adding this loss improves formant fidelity relative to the reference, and the improvement increases with $F_0$. Second, a matched loss built on unweighted linear prediction or on cepstral smoothing does not show the same pitch-dependent gain. Third, the same estimator used as a metric shows over-smoothing in current vocoders that mel-cepstral distortion understates for female and falsetto voices.
Research Program
The core object is weighted linear prediction with a fixed weight sequence. For a frame $x_n$, the filter coefficients $a_k$ of order $p$ minimize
$$E = \sum_n w_n \Big( x_n - \sum_{k=1}^{p} a_k x_{n-k} \Big)^2 ,$$
where $w_n$ is the attenuated-main-excitation (AME) weighting from Quasi closed phase analysis. This weighting sets a small constant $d$ around each glottal closure instant and through the open phase, and a high weight during the closed phase. Quasi-closed-phase (QCP) analysis is the method that uses it. For fixed $w_n$, the coefficients solve a $p \times p$ linear system whose entries are weighted autocovariances of $x$. The solution is therefore a differentiable function of the waveform. Ordinary automatic differentiation through a batched linear solve is enough, with a small ridge term for conditioning.
The weights are computed once from the reference recording, using closure instants detected on the real audio. They are then applied unchanged to both the reference frame and the time-aligned generated frame. This is possible because vocoders are trained by copy-synthesis from the reference's own acoustic features. The generator therefore never has to produce closure instants for the loss to be defined.
The loss compares log envelopes:
$$\mathcal{L}_{\text{QCP}} = \frac{1}{K}\sum_{\omega} \Big| \log |A_{\hat{x}}(e^{j\omega})| - \log |A_{x}(e^{j\omega})| \Big| .$$
Here $A_x$ and $A_{\hat x}$ are the inverse-filter polynomials fitted to the real and generated frames, and the sum runs over $K$ frequency points. Comparing the inverse filters rather than the all-pole responses means unstable poles cannot produce infinite values. This sidesteps the stability issue that Stabilised weighted linear prediction solves algebraically. The loss is applied only to voiced frames.
Two variants address the known fragility of QCP, which is its dependence on accurate closure instants. The first replaces AME with the short-time-energy weighting of stabilised weighted linear prediction (SWLP). That weighting needs no closure detection, and with a short window it behaves close to a smooth, harmonic-insensitive envelope estimator. The second is a frequency-domain weighting in the spirit of the size-shape-image weighting of Irino et al.. There, channels dominated by low harmonics are down-weighted. That weighting brought their speaker-size model's just-noticeable differences from over 16% to about 5%.
The program has three steps.
- Validate the estimator as a measuring instrument.
- Compute QCP, SWLP, plain LP and cepstral envelopes on the 63,750 physically modelled vowels used in the QCP paper, where true resonances are known, and on the hand-checked tracks of the Vocal Tract Resonance (VTR) database.
- Apply controlled spectral blurring to real speech and confirm that a QCP-based "over-smoothing index" rises monotonically with the amount of blur. The index combines formant bandwidth inflation and peak-to-valley loss relative to the reference.
- Audit existing vocoders with this index, binned by $F_0$.
- Train with the loss and measure the effect.
Baselines
The naive reference point is the vocoder trained with its published losses and nothing added. This isolates whether any envelope term helps at all.
The strongest existing competitor for speech is HiFi-GAN v1. It combines multi-period and multi-scale discriminators with a mel loss, and it is the reference that APNet reports against: MOS 3.99 on LJSpeech at 16 kHz, against 3.93 for APNet. UnivNet is arguably stronger on fine time-frequency texture because of its multi-resolution spectrogram discriminator. However, no reported MOS result was found on Lacuna for UnivNet; the file says only that it scored best among the GAN vocoders it was compared with. We therefore run it ourselves rather than quote a number.
For singing, the strongest system in the file is SingGAN. It reports MOS 4.05 against HiFi-GAN's 3.82 and ground truth's 4.25. Its explicit $F_0$-driven harmonic source makes it the hardest test of whether our loss adds anything beyond a good source model.
Each baseline is retrained from the public recipe on identical data splits. We use the same number of steps and the same hyperparameter search budget for the weight of the added loss term. The added term is the only change.
Two ablations isolate the key ingredient, which is closed-phase weighting rather than "some envelope loss."
- The identical loss computed with unweighted linear prediction ($w_n = 1$).
- A log-envelope loss from cepstral liftering of the STFT, which is a smooth, harmonic-agnostic envelope.
Beating a baseline means a statistically significant reduction in formant error relative to the reference in the $F_0 > 200$ Hz bins, with MUSHRA or MOS not worse at the 95% level. Beating the ablations means the QCP loss's advantage over them grows across $F_0$ bins, not merely that it is lower overall.
Why not just…?
Why not just rely on multi-resolution discriminators? This is the main threat. UnivNet's spectrogram discriminator and XiaoiceSing 2's patch discriminator were built precisely to restore sharp time-frequency detail. XiaoiceSing 2 reports 4.23 MOS against 4.27 for ground truth on an internal Mandarin set. Two points cut the other way. Discriminators still optimize a distribution-level signal that the file describes as biased toward dominant low-frequency energy. SingGAN reports that HiFi-GAN's discriminators fail on sustained high phonations. Our experiment settles the question directly: we add the loss on top of UnivNet. If the formant error in high-$F_0$ bins does not move, the discriminator already captures what we target, and we will say so.
Why not just use a diffusion vocoder? Diffusion avoids regression averaging. It does so at a large cost: DiffSVC uses 100 steps, and in VocBench WaveGrad's CPU real-time factor is 9.858, against 0.029 for MelGAN. Our loss is used only during training and adds no inference cost. We include a diffusion vocoder in the audit as a quality ceiling. It is not a competitor at equal latency.
Why not just feed better envelopes to the generator, as GlottDNN and LPCNet do? GlottDNN already uses QCP offline to drive its synthesis filter. LPCNet uses unweighted prediction and reports roughness between harmonics. Swapping in QCP coefficients changes the conditioning of a source-filter vocoder. It does nothing for the mel-conditioned GAN vocoders that dominate practice. It also leaves their losses uniform. We test the swap as a side experiment inside LPCNet. If it closes the high-pitch gap on its own, the conclusion is that the conditioning, not the loss, is where harmonic bias matters.
Why not just add a cepstral or mel-cepstral envelope loss? That is the second ablation. Cepstral smoothing is exactly the uniform, harmonic-blind smoothing that Kawahara et al. show smears formants at high pitch. If it performs as well as QCP, the proposal's specific mechanism is refuted.
Proposed Experiments
Estimator validation. On the synthetic vowel set and VTR, we report formant detection rate and formant estimation error per $F_0$ bin for each envelope estimator. The anchor is TVQCP-L1's reported 91.0% detection rate and 5.7% error on F3 from Gowda and Alku. Our QCP implementation should come close to that analysis-side performance before we trust it as a loss.
Vocoder audit and training. We train on LJSpeech, a single female speaker, and LibriTTS and VCTK, which are multi-speaker, using the VocBench splits. For high-register singing we use GTSinger (resource GTSinger), using its phoneme-level falsetto and breathy labels to form subsets.
We report the following:
- formant error relative to the reference, measured with an independent estimator (TVQCP) so the metric is not the training loss;
- the over-smoothing index;
- mel-cepstral distortion;
- $F_0$ error;
- MUSHRA by $F_0$ bin and by singing technique.
Decisive checks. Three results decide the outcome.
- Pitch dependence. The QCP loss's improvement over the unweighted-LP and cepstral ablations should grow from male speech to female speech to falsetto. A flat gain would mean we merely added a useful envelope term.
- Robustness to closure-instant errors. We perturb the closure instants by ±0.5 to 2 ms and check whether the SWLP variant keeps most of the gain without closure detection.
- Breathy voice. Breathy subsets, which have no true closed phase, should show little or no gain. This is a negative control that ties the effect to the physics rather than to extra supervision.
References
- Beyond bandlimited sampling of speech spectral envelope imposed by the harmonic structure of voiced sounds
- Flow-Based Unconstrained Lip to Speech Generation
- Quasi closed phase analysis for glottal inverse filtering
- Stabilised weighted linear prediction - a robust all-pole method for speech processing
- An Auditory Model of Speaker Size Perception for Voiced Speech Sounds
- APNet: An All-Frame-Level Neural Vocoder Incorporating Direct Prediction of Amplitude and Phase Spectra
- UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
- SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation
- Xiaoicesing 2: A High-Fidelity Singing Voice Synthesizer Based on Generative Adversarial Network
- Vocbench: A Neural Vocoder Benchmark for Speech Synthesis
- GlottDNN - A Full-Band Glottal Vocoder for Statistical Parametric Speech Synthesis
- LPCNET: Improving Neural Speech Synthesis through Linear Prediction
- Time-Varying Quasi-Closed-Phase Weighted Linear Prediction Analysis of Speech for Accurate Formant Detection and Tracking
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks