Hybrid text-to-speech systems build sentences by stitching together short pieces of real recorded speech. A statistical model first predicts a smooth acoustic "blueprint" for the sentence: pitch, spectral envelope and durations. A search then picks recorded pieces, called units, that match it. The weak point is that every recorded unit still carries traces of the sounds that surrounded it when it was recorded. A vowel cut from "ba" has its onset shaped by the /b/, and it keeps that shape when it is pasted after a /k/. Current systems score a candidate unit by its raw distance to the blueprint. That single number mixes two different things. One is distortion that the model itself can predict and partly remove, which comes from moving the unit into a new context. The other is the unit's own idiosyncrasy, which is exactly the natural detail that makes concatenation worth doing.
Particle physicists face the same problem when they correct for initial state radiation. In that effect, an incoming electron emits a photon before it collides, so the measured cross section is the true ("Born") one smeared by a known radiation kernel. The standard practice does not invert the data blindly. It splits the observation into a part computable from theory, which is corrected with a model-derived factor, and a part that is actually measured. It then treats the model-dependence of that factor as a systematic uncertainty, estimated by varying the model. We propose to carry this discipline over to unit selection. The acoustic model predicts the trajectory of each candidate both in its original recording context and in its new target context. The difference is a per-frame, model-computable "transplant correction." Part of it is applied to the unit's pitch and spectrum within an edit budget. Only the remaining residual is scored as mismatch. The spread of the correction across an ensemble of acoustic models then decides when a unit is too uncertain to use and the system should fall back to generated speech.
Core Claim
Let $x_u(f)$ be the acoustic parameters (mel-cepstrum, log-$F_0$, aperiodicity) of candidate unit $u$ at frame $f$. Let $\mu_\theta(f; c)$ be the trajectory that acoustic model $\theta$ predicts for that phone in linguistic context $c$. The usual target cost compares $x_u$ directly to $\mu_\theta(f; c_{\text{tgt}})$. We split that difference into two parts:
$$x_u - \mu_\theta(c_{\text{tgt}}) = \underbrace{\big(x_u - \mu_\theta(c_{\text{src}}(u))\big)}_{r_u:\ \text{unit's own residual}} + \underbrace{\big(\mu_\theta(c_{\text{src}}(u)) - \mu_\theta(c_{\text{tgt}})\big)}_{-\Delta_u:\ \text{predicted transplant shift}}$$
Here $c_{\text{src}}(u)$ is the context the unit was recorded in. The shift $\Delta_u$ is "calculable," much as the radiator kernel is calculable in R value measurements for e+e− annihilation at 2.60, 3.07 and 3.65 GeV, where the observed cross section is linked to the Born one through an $\mathcal{O}(\alpha^3)$ correction. We apply $\Delta_u$ to the unit up to a per-frame edit budget $b(f)$ using pitch and envelope modification. The selection cost is then
$$C(u) = \|r_u\|^2_{\Sigma} + \lambda_1 \|\max(0, |\Delta_u| - b)\|^2 + \lambda_2\, \mathrm{Var}_{\theta \in \Theta}[\Delta_u],$$
where $\Sigma$ is the model's predicted variance, the second term penalizes any shift that cannot be applied within budget, and the third term is the variance of the shift across an ensemble $\Theta$ of acoustic models. That third term plays the role of the model-dependence systematic.
The claim is that this decomposition does three things the existing systems lack. It gives a data-derived, frame-wise correction profile in place of a hand-tuned boundary window. The profile should concentrate at unit edges, because that is where context matters, but it is not forced to. It covers $F_0$ and aperiodicity, not only the spectrum. And it turns the decision to fall back to generated speech into an uncertainty criterion instead of a fixed threshold.
Relation to Prior Work
The hybrid line supplies the machinery we reuse. CMU Blizzard 2007 uses a predicted trajectory as a soft Euclidean target cost. HMM trajectory tiling tiles units along a trajectory trained with minimum generation error and aligns joins by normalized cross-correlation. Frame-sized HMM unit selection casts selection as maximum likelihood over 5 ms frames. All three score raw unit-to-target distance. None of them separates correctable context shift from the unit's own residual.
The closest work on correction is Local minimum generation error. It adapts model means toward the selected units at synthesis time, using a sigmoid frame weight $w(f)$ that is low at joins. Its corrections run in the opposite direction from ours: it moves the model toward the units, while we move the units toward the context the model predicts. It also leaves $F_0$ and aperiodicity untouched, a limitation the paper itself reports, and its weight function is set by hand. In our method the profile comes from $\Delta_u$.
UnitNet learns unit embeddings for its target and concatenation costs, and generates a phone with its neural branch whenever the best cost exceeds a threshold. It retrieves about 97% of phones and reaches a real-time factor of 0.484. Our uncertainty term is meant to replace that fixed threshold. The Siri on-device system constrains the design: real systems must preselect down to roughly 100 candidates. Our cost is therefore computed only on preselected candidates, and the source-context predictions $\mu_\theta(c_{\text{src}}(u))$ are cached offline per unit.
The physics papers supply the methodological lens rather than any code. Two lessons in particular shape the design. First, unfolding is model-dependent, and in the R value measurement a roughly 3.5% systematic dominated a sub-1% statistical error. That lesson motivates the ensemble variance term and the requirement that we report how much selections change under model variation. Second, correcting only the leading-order effect can produce qualitatively wrong conclusions. The ³He target-normal asymmetry is the example: the single-photon Born picture predicts zero, yet the measured neutron asymmetry is $(-1.09 \pm 0.38)\times10^{-2}$. That lesson is why we refuse to correct only the spectrum and ignore excitation.
We deliberately do not deconvolve waveforms to recover "bare" phonemes, as the research file's Gap 1 suggests. That inversion is ill-posed and would destroy the natural detail that concatenation exists to preserve.
Research Program
We first train a duration-explicit neural acoustic model, along with an ensemble of 5 variants differing in seed and data subset. Each predicts mean and variance for mel-cepstrum, log-$F_0$, band aperiodicity and duration given full linguistic context.
For every database unit we cache $\mu_\theta(c_{\text{src}}(u))$, time-aligned to the unit's own forced-aligned frames. At synthesis we compute $\Delta_u$ for the preselected candidates. The pitch component of $\Delta_u$ is applied with pitch-synchronous overlap-add. The spectral component is applied as a minimum-phase envelope filter, interpolated per frame. The budget $b(f)$ is calibrated once from listening tests on how much pitch and envelope modification remains inaudible. The Viterbi search uses $C(u)$ together with a standard join cost computed on the corrected parameters.
The analysis component asks three questions. Where in time does $|\Delta_u|$ concentrate? Does it reproduce or contradict LMGE's sigmoid? How much of the raw target-cost variance is explained by $\Delta_u$ versus $r_u$?
Baselines
The naive reference point is trajectory-guided selection with an uncorrected Euclidean target cost, as in CMU Blizzard 2007. That paper reports a mean opinion score (MOS) of 3.38 against 2.98 for its plain unit selection system on its 8-hour database. We reimplement it with our own acoustic model, so the only difference from our method is the cost.
The strongest classical competitor is HMM trajectory tiling. It reports a MOS 0.9 points above conventional unit selection on Blizzard 2010 Corpus B, and that number transfers directly because we use the same corpus. We will run HTT with normalized cross-correlation joins using the same neural trajectory.
The strongest modern competitor is UnitNet, with learned unit embeddings and threshold fallback. No absolute MOS for UnitNet was found on Lacuna. Its reported operating point is about 97% retrieval at a real-time factor of 0.484 on a 17.5-hour Chinese corpus. We will retrain it on our corpora and tune its threshold on the same development set and the same number of listening-test configurations as our $\lambda$ values.
We also include LMGE-style synthesis-time adaptation with the sigmoid window, and a purely neural parametric system with a neural vocoder as the quality ceiling for smoothness. All systems share preselection, corpora and vocoder where a vocoder is used, and receive equal hyperparameter search budgets.
Four ablations isolate the key ingredient. The first scores $r_u$ but applies no edit. The second applies the edit but scores raw distance. The third corrects spectrum only, mimicking LMGE's coverage. The fourth drops the ensemble term in favor of a fixed fallback threshold.
Beating the baselines means two things. Against HTT and UnitNet, we need a statistically significant MOS and AB-preference gain at equal or lower fallback rate. On the speaker-similarity score, we need to be no worse than plain selection.
Why not just…?
Why not just put context-mismatch terms in the linguistic target cost, as classical unit selection does? This is the most serious threat. Classical target costs already penalize choosing a unit from a different phonetic neighborhood. The difference is that such terms are categorical and uniform over a unit. $\Delta_u$ is acoustic, frame-wise and graded. It tells the system that a /b/-to-/k/ swap barely moves this vowel's pitch but shifts its first 20 ms of envelope, and that the shift is correctable. To settle the question, we add a tuned linguistic-mismatch cost to the naive baseline and to the "score $r_u$, no edit" ablation. If that baseline matches our full system, the edit and decomposition add nothing and we would say so.
Why not just let a learned embedding cost absorb context, as UnitNet does? UnitNet's embeddings may well encode source context implicitly. However, a scalar cost cannot also prescribe an edit, and its fallback threshold has no notion of model uncertainty. The UnitNet-with-our-edit hybrid condition tests whether the gain comes from the edit alone.
Why not just use a neural vocoder end to end? In the USTC Blizzard 2017 system, a WaveNet-vocoded system scored only 3.1 out of 5 on similarity against 3.6 on naturalness. UnitNet's pure parametric path ran at a real-time factor of 1.360. Our method targets the regime of speaker fidelity and low compute. We nonetheless include the neural system so the size of the remaining gap is visible.
Why not just smooth the joins? Cross-fading and cross-correlation alignment fix the waveform offset at the join. They do not fix pitch or envelope shifts that extend inside the unit. The spectrum-only and sigmoid-window ablations measure exactly this difference.
Proposed Experiments
We use the Blizzard Challenge 2010 Corpus A (1 hour) and Corpus B (4.5 hours) of British English, which differ in coverage, and the 6.5-hour expressive audiobook corpus from Blizzard 2017, where context-driven pitch variation is large.
Metrics are MOS and AB preference from at least 20 listeners and speaker similarity against natural reference. Objective measures are mel-cepstral distortion and $F_0$ jump magnitude across joins, fallback rate, and real-time factor. We also report a "systematic" score: the fraction of selected units that change when the acoustic model is swapped within the ensemble.
Three outcomes are decisive. First, the gain over HTT and UnitNet should be larger on Corpus A than on Corpus B, because sparser coverage forces more context transplants. A flat or reversed pattern would undercut the mechanism. Second, splitting test joins by $\|\Delta_u\|$, the objective and perceptual gains should concentrate where source and target contexts differ most. Third, at a matched fallback rate, uncertainty-based fallback should beat threshold fallback, and the units it rejects should have higher listener-rated glitch rates. If the linguistic-mismatch baseline closes the gap on all three, we conclude that the decomposition is a reinterpretation rather than an improvement.
References
- R value measurements for <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.gif" overflow="scroll"><mml:msup><mml:mi>e</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>e</mml:mi><mml:mo>−</mml:mo></mml:msup></mml:math> annihilation at 2.60, 3.07 and 3.65 GeV
- CMU Blizzard 2007: a hybrid acoustic unit selection system from statistically predicted parameters
- An HMM trajectory tiling (HTT) approach to high quality TTS
- HMM-based unit selection using frame sized speech segments
- Local minimum generation error criterion for hybrid HMM speech synthesis
- UnitNet-Based Hybrid Speech Synthesis
- Siri On-Device Deep Learning-Guided Unit Selection Text-to-Speech System
- Measurement of the Target-Normal Single-Spin Asymmetry in Deep-Inelastic Scattering from the Reaction <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msup><mml:mrow><mml:mmultiscripts><mml:mrow><mml:mi>He</mml:mi></mml:mrow><mml:mprescripts/><mml:none/><mml:mrow><mml:mn>3</mml:mn></mml:mrow></mml:mmultiscripts></mml:mrow><mml:mrow><mml:mo>↑</mml:mo></mml:mrow></mml:msup><mml:mo stretchy="false">(</mml:mo><mml:mrow><mml:mi>e</mml:mi><mml:mo>,</mml:mo></mml:mrow><mml:msup><mml:mrow><mml:mi>e</mml:mi></mml:mrow><mml:mrow><mml:mo>′</mml:mo></mml:mrow></mml:msup><mml:mo stretchy="false">)</mml:mo><mml:mi>X</mml:mi></mml:mrow></mml:math>
- The USTC System for Blizzard Challenge 2017