Self-Calibrating Pitch Targets for Expressive Speech Synthesis

Text-to-speech systems that predict pitch, duration and loudness explicitly are trained to imitate pitch curves that are not measured directly. They come from automatic pitch trackers such as Crepe or Praat. These trackers make errors, and the errors are not random. They tend to cluster in shouted, creaky, breathy or very high-pitched speech, which is exactly the speech that carries emotion: the tracker jumps an octave, calls a voiced frame silent, or clips an extreme value. A model trained on such curves learns a flattened picture of expressive speech before any loss function has a chance to average it away. Remote sensing has dealt with this kind of problem for decades. When a radar satellite has a timing or atmospheric bias, more images do not make it disappear. Engineers remove it by demanding that the same ground point, seen from several known viewpoints, land at the same coordinates, and by solving jointly for the bias and the true positions without any surveyed ground markers.

We propose to transfer that recipe to speech. Each pitch tracker is treated as a biased "sensor." The true pitch of each audio frame is the unknown "ground point." Known pitch transformations of the same audio, such as resampling by an exact ratio, serve as the extra "viewpoints" whose geometry we know. Solving one joint least-squares problem across trackers and transformed copies gives corrected pitch targets and a per-tracker bias model, with no hand-labelled pitch. The bet is that part of the flatness of explicit-prosody speech synthesis is a measurement artifact. Removing it should make deterministic and diffusion prosody predictors more expressive at no extra inference cost, and should also make pitch-based evaluation metrics fairer to expressive systems.

Core Claim

The standard account of flat synthetic speech is regression to the mean. Speech prosody is one-to-many, and an L1 or L2 predictor outputs the average. Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model responds by replacing the regression predictor with a diffusion model, a generator that learns to turn noise into samples step by step. That change lowered pitch Jensen–Shannon divergence from 0.199 to 0.085, but it required 500 sampling steps. Notably, that work extracts its pitch targets with Crepe and treats them as truth.

Our claim is complementary and testable. Pitch-tracker error has a systematic component that grows with vocal effort and unusual voice quality. This component compresses the pitch range and the voiced regions of high-arousal targets. Removing it with a multi-view self-calibration should help in a specific pattern. The gains should be largest for emotions such as anger, surprise and happiness, small for neutral speech, and present for both deterministic and diffusion predictors.

The closest work in the file does something different. Controllable Neural Prosody Synthesis fights pitch leakage inside the vocoder with a learned bottleneck. Boosting Multi-Speaker Expressive Speech Synthesis with Semi-Supervised Contrastive Learning separates speaker, style and emotion in embedding space. The research file itself notes that this work does not handle continuous measurement errors from pitch trackers. Neither line of work corrects the training targets themselves.

The Method, Borrowed from Bundle Adjustment

The method role is played by Geometric Self-Calibration of YaoGan-13 Images Using Multiple Overlapping Images and Block Adjustment without GCPs for Chinese Spaceborne SAR GF-3 Imagery. These papers write error equations linking sensor biases to observed positions of shared ground points. They then minimize intersection residuals over many overlapping passes, without ground control points. We reuse the structure and swap the physics.

For frame $t$ of an utterance, let $f_t$ be the unknown true pitch. Let tracker $k$ (Crepe, Praat autocorrelation, and a probabilistic YIN variant) be run on a copy of the audio resampled by a known ratio $\alpha$. Resampling scales the true pitch exactly by $\alpha$, and it also shifts formants and spectral tilt. The observation model is

$$\log \hat f^{(\alpha)}_{k,t} = \log f_t + \log \alpha + b_k\big(z^{(\alpha)}_t\big) + o_{k,t}\log 2 + \epsilon_{k,t},$$

where $\hat f^{(\alpha)}_{k,t}$ is the tracker's reading, $b_k$ is a smooth bias function of acoustic descriptors $z_t$ (harmonics-to-noise ratio, spectral tilt, frame energy, local register), $o_{k,t}\in\{-1,0,1\}$ is a discrete octave-error indicator, and $\epsilon_{k,t}$ is random noise. Voicing decisions get an analogous per-tracker confusion model.

We fit by alternating two steps. The first is a robust least-squares update of $\log f_t$ and the parameters of $b_k$. The second is a Viterbi-style update of the octave indicators under a smoothness prior on $f_t$. This mirrors the iterative adjustment in the YaoGan-13 pipeline.

The known ratios $\alpha$ do the work that multiple look angles do in SAR. If all trackers shared the same bias and we only compared trackers to each other, a common-mode bias would be invisible. Resampling changes $z_t$ while fixing the relationship between true pitch values, and that is what makes $b_k$ identifiable. The constraining lesson from the SAR literature also carries over. YaoGan-13 reports that decoupling depends on enough multi-angle redundancy and fails when the normal equations become ill-conditioned. We will therefore report the condition number of the problem as a function of the number and spread of ratios.

A second borrowed idea comes from Decimeter-Level Geolocation Accuracy Updated by a Parametric Tropospheric Model with GF-3. There, removing a physically modelled delay before adjustment lowered zenith delay RMSE from 38.4 mm to 10.9 mm. By analogy, we remove speaker-level register offsets (mean log pitch per speaker and session) before solving for tracker biases. This keeps register differences from being misread as tracker bias.

Research Program

The work has three stages.

The first stage is calibration and verification. We need independent check points, the analog of corner reflectors. We will build two sets. The first is resynthesized expressive utterances from the Emotional Voice Conversion: Theory, Databases and ESD corpus, generated with an imposed, exactly known pitch contour. The second is a few hundred hand-annotated frames drawn from high-arousal regions. Neither set is used in fitting. A known weakness is that resynthesized audio may not reproduce real creak or breathiness, so the human-annotated frames are the more trustworthy test.

The second stage is target replacement. We retrain prosody predictors on corrected pitch while keeping everything else fixed.

The third stage is an evaluation audit. The TTSDS2 prosody factor and pitch Jensen–Shannon divergence are both computed from extracted pitch. We will recompute them with calibrated pitch to see whether tracker bias also makes current metrics under-reward expressive systems.

ESD is the main corpus for both scientific and practical reasons. It contains 350 parallel sentences per speaker in five emotions, so the same text and speaker appear across arousal levels. This lets us separate emotion-dependent tracker bias from text effects. The English half has 10 speakers and is recorded above 20 dB signal-to-noise ratio, which limits how much channel noise is mixed into the bias.

Baselines

The naive reference point is a FastSpeech-2-style model with deterministic L1 pitch, energy and duration predictors, trained on raw Crepe targets. This is the system the field calls over-smoothed. Beating it is necessary but proves little.

The strongest existing competitor is the phoneme-level diffusion prosody predictor of Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model. On its private Mandarin audiobook corpus it reports pitch JS divergence of 0.085, against 0.199 for the deterministic predictor, and expressiveness MOS of 3.98 against 3.58. These numbers are not on ESD, and there is no reported result on Lacuna for that predictor on ESD. We will reimplement it on ESD English with the same acoustic model, vocoder, data split, GPU budget and number of tuning runs as every other arm.

The comparison is a two-by-two: deterministic or diffusion predictor, crossed with raw or calibrated targets. Two outcomes would count as success. The first is a deterministic predictor on calibrated targets closing at least half of the JS-divergence gap between the deterministic and diffusion predictors on raw targets, using one forward pass instead of 500 steps. The second is the diffusion predictor on calibrated targets beating its raw-target version on high-arousal emotions.

Three ablations isolate the key ingredient. The first is a median vote across the same three trackers with standard octave-jump smoothing, with no bias model and no resampled views. The second is our joint solve using tracker agreement only, without resampled copies. The third is our full solve without the speaker-register pre-correction. The second ablation tests the central claim that the known-transform views are what identify the bias.

We also report Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis-style pitch-curve correlation for reference-guided synthesis. Daft-Exprt reports a correlation of 0.43 on its own benchmark. We use this only as context, not as a directly comparable number.

Why not just…?

Why not just use a better single tracker, or a median of trackers with octave-fix heuristics? This is the real threat, and the median-vote ablation exists to settle it. Voting removes errors on which the trackers disagree. It cannot remove a bias they share, and different neural and autocorrelation trackers plausibly share bias on creaky or breathy voice, because all of them depend on weak periodicity. If the median baseline matches our solve on the human-annotated check frames and on downstream expressiveness, the calibration machinery is unnecessary and we will say so.

Why not just rely on a diffusion predictor, since it models the full distribution anyway? A generative model fits whatever distribution it is shown. If the targets are compressed in expressive regions, the diffusion predictor faithfully reproduces that compression. The 2×2 design answers this directly. If calibrated targets help the deterministic predictor but not the diffusion one, our claim narrows to "a cheap substitute for diffusion" rather than "a fix for both."

Why not just drop explicit pitch altogether, given that end-to-end models such as E2-TTS and MaskGCT score within 0.05 of human speech on TTSDS2? This is a fair limit on scope. TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems does report that result. Our contribution matters for systems that keep explicit pitch for control, such as Daft-Exprt, FastPitchFormant: Source-Filter Based Decomposed Modeling for Speech Synthesis, the controllable model of Morrison et al., and pitch-conditioned conversion such as Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model, which uses ESD. All of them inherit tracker errors. The evaluation audit applies to end-to-end systems too, because their prosody scores still pass through a pitch tracker.

Why not just train with a Wasserstein loss on pitch distributions? That is the research file's Gap 4. A distribution-matching loss would still match the distribution of biased targets. It is orthogonal to our proposal and could be stacked on top later.

Proposed Experiments and Decisive Checks

The first decisive check happens before any speech synthesis. On the held-out check frames, calibrated pitch must reduce gross pitch error and voicing-decision error relative to every single tracker and the median vote, with the largest reduction in angry and surprised ESD frames. It must also do so only when resampled views are included, and the condition number of the solve should explain where it fails. If raw tracker error on ESD shows no dependence on emotion, the premise is false and the project stops there.

The second check is downstream. We train all arms on ESD English and evaluate on held-out sentences. The measures are per-emotion pitch JS divergence against calibrated real pitch, pitch range, and the TTSDS2 prosody and intelligibility factors. We also test emotion recognizability with an ESD emotion classifier, which reaches 89% on real English speech. Finally, we run a listening test for expressiveness MOS and an AB preference test, under the same protocol for every arm. The prediction is an interaction effect: improvements concentrated in high-arousal emotions, with word error rate unchanged.

The third check is the evaluation audit. We rescore the deterministic and diffusion arms with both raw and calibrated pitch inside the prosody metrics. If the ranking or the size of the gap between them changes, then current pitch-based metrics carry the same bias we are correcting in training, and that would be a finding in its own right.