Speech synthesizers trained to predict acoustic features frame by frame tend to output an average of all plausible renditions. The result sounds muffled and monotone, a failure called over-smoothing. The classical fix is global variance (GV) constraints, which force the generated feature trajectory to swing as widely over an utterance as natural speech does. Toda and Tokuda wrote this as a joint objective and solved it with a local Newton-type search started from the over-smoothed trajectory. Because the objective is non-convex, the field has since treated the search as heuristic. Nobody has asked whether it finds the best trajectory, or whether it depends on where it starts. The registration literature, which fights the same kind of starting-point sensitivity when aligning 3D point clouds, would suggest continuation schemes or branch-and-bound search.
This note argues that neither is needed. For the standard diagonal-GV objective, the non-convexity is hidden convexity of a well-known kind. Generating a trajectory under a variance target is a quadratic problem with a single quadratic constraint, the same structure as the trust-region subproblem in numerical optimization. Its global optimum is a single-parameter family of ordinary maximum-likelihood solutions with a weakened precision matrix. The whole problem therefore reduces to a one-dimensional scan over a scalar multiplier, where each step is a banded linear solve. This yields a certified global solver. It also allows an exact test of how often the classical local search misses the optimum. And it gives an analytic answer to which temporal fluctuations GV can and cannot restore, which connects GV to the local over-smoothing that mixture models target.
Core Claim
Fix one feature dimension, for example one mel-cepstral coefficient. Let $c\in\mathbb{R}^T$ be its static trajectory over $T$ frames. The maximum-likelihood (ML) term with static, delta and delta-delta windows is the concave quadratic $-\tfrac12 c^\top R c + r^\top c$. Here $R=W^\top U^{-1}W$ is the banded trajectory precision built from window matrix $W$ and the frame-wise model precisions $U^{-1}$, and $r=W^\top U^{-1}\mu$ collects the predicted means $\mu$. The utterance variance is $v(c)=\tfrac1T c^\top P c$, where $P=I-\tfrac1T\mathbf{1}\mathbf{1}^\top$ removes the mean. With a diagonal Gaussian GV model of mean $\mu_v$ and variance $\sigma_v^2$, the objective of Toda & Tokuda is
$$J(c) = -\tfrac12 c^\top R c + r^\top c - \frac{\omega}{2\sigma_v^2}\big(v(c)-\mu_v\big)^2 ,$$
a quartic in $c$ with weight $\omega$. Split $J$ by the achieved variance $s=v(c)$. For each $s$, the inner problem maximizes a concave quadratic over the set $c^\top P c = Ts$. By the S-lemma, this problem has zero duality gap. Its global maximizer satisfies
$$(R+\nu P)\,c = r,\qquad R+\nu P \succeq 0,$$
for a scalar multiplier $\nu$. The branch $\nu>\nu_0$, where $R+\nu P$ stays positive semidefinite, maps monotonically onto all achievable variances $s>0$. So the global optimum of $J$ is $c(\nu)=(R+\nu P)^{-1}r$ for the best $\nu$ on that branch, and it can be found by scanning one scalar.
Each evaluation of $c(\nu)$ is a banded solve plus a rank-one correction, costing $O(T)$. Fifty to a hundred evaluations, plus local refinement, give a certificate that is exact up to scan resolution. The rare degenerate case, where $r$ has no component along the boundary eigenvector, is handled by adding that eigenvector.
Two consequences matter.
First, starting-point sensitivity in GV generation becomes measurable. Problems of this type have at most one non-global local optimum. A Newton search can land there if it crosses the branch point $\nu_0$, and we can now detect when it does.
Second, the solution has a physical reading. GV sharpening is exactly ML generation with the model's confidence in centered deviations uniformly reduced by $|\nu|$. The added variance therefore flows mostly into the directions where $R$ is weakest, which the delta windows make slowly varying. The prediction is that GV mainly restores slow, low-confidence modulations and restores little fast spectral-temporal detail. That is a precise, testable account of why global constraints and local mixture models such as TVC-GMM fix different parts of over-smoothing.
Research Program
Stage 1: solver and certificate. We will implement the $\nu$-scan solver for diagonal GV and verify it against brute-force search on short synthetic trajectories. We will then run it alongside Toda and Tokuda's diagonal-Hessian Newton update, started from the ML trajectory as in the original paper, on every test utterance and every feature dimension. For each case we record the objective gap, whether the local solution sits on the invalid branch $\nu<\nu_0$, the iteration count, and wall-clock time.
Stage 2: which fluctuations GV restores. We will decompose the difference between the GV-optimal and ML trajectories along the eigenvectors of $R$ restricted to the mean-removed subspace. We will then compare the resulting modulation-frequency profile against natural speech. The goal is to show quantitatively how much of the natural-versus-generated gap lies at temporal frequencies GV cannot reach.
Stage 3: differentiable GV layer. Because the optimum depends on one scalar $\nu^*$, the implicit function theorem gives gradients through the solver cheaply. We will place this exact layer on top of a neural acoustic model that outputs static and dynamic means and precisions. This replaces the approximate, pretraining-dependent training of Hosaka et al., whose trajectory-GV loss for voice conversion was optimized by SGD after frame-level MSE initialization, with sensitivity to that initialization left unstudied.
Stage 4: context-dependent targets. An exact solver makes any variance target cheap to enforce. This lets us attack the "fixed target statistics" limitation with a small regressor that predicts $\mu_v$ per utterance from text or source features.
The full-covariance GV model couples dimensions. Given a variance vector, the inner problems still decouple, so only a $D$-dimensional outer problem is non-convex. Here, and only here, branch-and-bound in the style of Go-ICP becomes relevant. We will treat that as an exploratory extension and report bounds rather than claim tractability.
Relation to Prior Work
The registration papers motivate the question rather than supply the method. Go-ICP certifies global rigid alignment by branch-and-bound over transformation space. Graduated non-convexity anneals a smoothed objective toward the true one. FMR, FilterReg and MMD-Reg widen the range of starting points that converge by replacing nearest-neighbour matching with smooth feature, filtering or kernel costs. All four treat non-convexity as something to search around. We instead exploit structure that removes the search. The closest analogy is Ho et al., who solve registration algebraically through moment matching.
The research file's own suggested gap, continuation from an over-smoothed objective toward full GV, is the obvious idea at this intersection. We include it as a baseline because the exact solver makes it unnecessary if our analysis holds.
On the speech side, Saito et al. replace hand-crafted GV with adversarial training and find that Wasserstein GANs work best. Takamichi et al. use moment matching to add sampling variance. Kögel et al. model local multimodality over adjacent time-frequency bins in FastSpeech 2. These address over-smoothing through distribution modelling. Our work sharpens the explicit-constraint route, which Minixhofer et al. indirectly motivate: they show diffusion models underperform MSE models around 100 hours of data, the low-resource regime where GV-style constraints remain the practical tool.
Baselines
The naive reference point is plain ML generation with no GV, $c=R^{-1}r$. It fixes the size of the over-smoothing gap every method must close. Next comes utterance-level post-processing variance scaling as used by Silén et al. and Xu et al.. It stretches each centered trajectory by $\sqrt{\mu_v/v_{\text{gen}}}$ and therefore hits the target variance exactly, but along a fixed direction. It gets the same GV statistics as our solver, so the comparison isolates the effect of choosing the optimal direction.
The strongest existing method on this exact objective is Toda and Tokuda's Newton-type update started from the ML trajectory. We run it with the original diagonal-Hessian scheme and a step-size sweep tuned on held-out utterances. We give it the same GV statistics and matched or larger wall-clock time. Toda and Tokuda report only qualitative naturalness gains; no reported result found on Lacuna for a numerical score from that paper. We also run two optimization baselines borrowed from registration: 20 random restarts of the Newton search, and a continuation schedule that anneals $\omega$ from zero in the spirit of graduated non-convexity. Both receive the same solve budget as our scan.
For the trained-model stage, the strongest competitor is the trajectory-GV network of Hosaka et al.. It reports a Global Variance Distance (GVD) of 0.349 against 0.378 for a frame-wise DNN with 450 ATR training sentences. Beating it means lower GVD and lower mel-cepstral distortion than 0.349 under the same data, network size and training epochs, together with a significant preference-test win. A W-GAN acoustic model following Saito et al. serves as the distribution-matching reference. Saito et al. give no numerical score; no reported result found on Lacuna for it.
The key ablation swaps the exact layer for the Newton layer inside the same training loop. This isolates whether exactness, rather than GV itself, matters.
Why not just…?
Why not just scale the variance after generation? Post-scaling reaches the target variance, but along the direction $c-\bar c$. Our analysis says the optimal direction is along the weak eigenvectors of $R$. On the objective, post-scaling is therefore provably suboptimal. The open question is whether listeners hear the difference. Hosaka et al.'s DMOS gains over post-filtering suggest they do, but that comparison also changed training. Our experiment holds GV statistics fixed and changes only the direction. If preference tests are at chance, we will report that post-scaling suffices at the variance level, and the contribution shifts to the analysis in Stage 2.
Why not just keep Toda and Tokuda's Newton search, which may already be near-optimal? This is the real threat. If the ML trajectory is usually on the valid branch, the local search may rarely miss. Stage 1 measures exactly this, per utterance. If misses are rare, the contribution becomes certification, removal of the step-size heuristic, and a cheap exact gradient for training, rather than quality gains. We state this outcome in advance.
Why not just use random restarts or continuation, as registration does? Neither yields a certificate. Our scan is effectively a one-dimensional exhaustive restart over the only coordinate that matters, at similar cost. The matched-budget comparison shows whether either ever finds a better objective value than the scan. By construction it should not.
Why not just use GANs or diffusion? These change the model rather than solve the GV objective, and Minixhofer et al. show diffusion models lose to MSE models near 100 hours of data. Our 10- and 450-sentence regimes are far smaller. We include the W-GAN reference so readers can see where exact GV stands against it, not to claim that GV replaces distribution modelling at scale.
Proposed Experiments
The primary testbed is the ATR Japanese database with 450 training and 53 evaluation sentences. This matches Toda and Tokuda, Hosaka et al. and Saito et al. It covers text-to-speech on one speaker and voice conversion from speaker m002 to m001 at 10 and 450 training sentences, which allows direct comparison with Hosaka's GVD figures.
For readers without ATR access, we replicate the solver study on LJSpeech and on VCTK voice-conversion pairs. We use an acoustic model with a mean-and-precision output head built on the FastSpeech 2 backbone of Kögel et al. On LibriTTS, that paper reports MOS 3.23 for MSE and 3.41 for TVC-GMM. We will test whether exact GV combined with TVC-GMM exceeds 3.41. This is where Stage 2 predicts the two methods are complementary.
Metrics are the certified objective gap and branch-violation rate, mel-cepstral distortion, GVD, Laplacian variance of spectrograms, modulation-spectrum distance per temporal-frequency band, runtime, and MOS/AB preference tests with at least 20 listeners. We also report fundamental frequency (F0) separately on voiced segments, given its known extraction-error problems.
Three results are decisive. The first is the fraction of utterances on which the local Newton search is certifiably suboptimal, and whether those utterances are audibly worse. The second is whether GV's restored variance concentrates in low modulation frequencies as predicted; a flat profile would falsify the interpretation. The third is whether the exact differentiable layer beats Hosaka's GVD of 0.349 at equal budget.
References
- Speech parameter generation algorithm considering global variance for HMM-based speech synthesis
- Voice Conversion Based on Trajectory Model Training of Neural Networks Considering Global Variance
- Go-ICP: Solving 3D Registration Efficiently and Globally Optimally
- Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection
- Feature-Metric Registration: A Fast Semi-Supervised Approach for Robust Point Cloud Registration Without Correspondences
- FilterReg: Robust and Efficient Probabilistic Point-Set Registration Using Gaussian Filter and Twist Parameterization
- Scalable and Differentiable Point-Cloud Registration Using Maximum Mean Discrepancy
- An algebraic approach to affine registration of point sets
- Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks
- Sampling-Based Speech Parameter Generation Using Moment-Matching Networks
- Towards Robust FastSpeech 2 by Modelling Residual Multimodality
- Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR
- Ways to implement global variance in statistical speech synthesis
- Dynamic noise aware training for speech enhancement based on deep neural networks