Text-to-speech systems are increasingly trained on "found" audio: podcasts, audiobooks and songs cleaned by automatic source separation (tools that pull one voice out of a mixture) and denoisers. These tools leave artifacts behind. They punch holes in the spectrum, smear phase, and leave faint "ghosts" of the background. Prior work shows that synthesis models learn to reproduce these artifacts in the audio they generate. This note proposes that the artifacts cause a second, less visible harm. They corrupt the pitch and energy contours that expressive models are trained to predict, and the corruption is worst for higher-pitched voices, whose widely spaced harmonics are easiest for a separation mask to delete. The result would be synthetic speech that is flatter and less expressive for women, a gap that looks like a demographic data problem but is partly a preprocessing problem.
The core idea is simple to state. First, measure whether separation artifacts shrink the pitch variation a model learns, and whether they shrink it more for female than for male speakers, while controlling separately for how many speakers of each gender the corpus contains. Second, if they do, fix the training targets instead of the model. A small pitch estimator combines complementary views of the signal: the damaged magnitude spectrum, an excitation-source feature from the vocal folds, and frozen self-supervised speech features. It is trained on pairs of separated and clean audio, and its repaired contours become the prosody targets for an ordinary expressive synthesizer.
Core Claim
The flattening of prosody in synthesis is usually blamed on the training objective. Deterministic losses average over the many valid ways a sentence can be spoken, and models regress to a flat mean (Style Tokens; CHiVE; non-autoregressive pitch contours show lower standard deviation than human ones in Integrating Discrete Word-Level Style Variations).
We claim there is a separate, upstream cause on separated data. Pitch trackers run on artifact-laden audio produce octave errors, spurious unvoiced frames and clipped excursions. Those errors concentrate at exactly the high-variance peaks and vowel onsets that carry expressiveness, and they concentrate in higher registers.
This yields a falsifiable prediction. On a corpus where women are the majority (AISHELL-3, 140 female vs. 26 male training speakers per Speaker Adaption with Intuitive Prosodic Features), artifact-induced flattening should still hit female voices harder. That result would separate register effects from minority-status effects, which prior gender-gap work cannot do because it studies only male-heavy corpora. The second claim is that repairing the targets with fused features recovers most of the lost pitch variation without any change to the synthesizer.
How This Relates to Prior Work
The motivating evidence comes from three places. DenoiSpeech shows that TTS decoders mimic enhancement distortions. It fixes this with a frame-level noise condition inside FastSpeech 2, but it leaves the deterministic pitch predictor untouched and never stratifies by gender. SingNet builds 3,000 hours of singing through separation and concedes that ghosting remains, but it offers no mechanism for suppressing it. LibriTTS-R restores audiobook speech generatively. It reports occasional hallucinated glitches and no gender-stratified evaluation.
Our proposal differs from all three in where it intervenes. We do not clean the waveform or condition the decoder on noise. We repair the prosody labels, and we measure the effect by register.
The method draws on the feature-fusion literature. Excitation-source evidence from the Hilbert envelope of the linear-prediction residual (the signal left after removing the vocal-tract filter, which tracks glottal pulses) localizes vowel onsets better than energy cues (Detection of vowel onset point events using excitation information). Phase statistics expose glottal source quality that magnitude spectra miss (Phase distortion statistics for glottal source analysis). Both are candidates for surviving a magnitude-domain separation mask. The choice of fusion operator is constrained by Attention Is Not Always the Answer, which found that addition or concatenation of MFCCs with frozen pretrained features beats cross-attention for voice activity detection. Pitch estimation is likewise a frame-level regression, so we default to concatenation and test attention only as an ablation.
Two papers constrain the design. Twists, Humps, and Pebbles finds that raw pitch and intensity do not explain ASR gender gaps, and that models which encode gender strongly show smaller gaps ($\rho = -0.65$). We take both findings seriously. The first concerns recognition on largely unprocessed audio; our pathway is corrupted training targets for a generator, which that study does not examine. The second argues against making the estimator gender-blind, so we give it a speaker embedding instead of removing gender information. Mellotron shows that explicit pitch conditioning is sensitive to extraction errors, and this is precisely the weakness our repaired targets address.
Research Program
Paired corpus. The first component is a paired corpus with known ground truth. We take clean LibriTTS, AISHELL-3 and VCTK utterances and mix them with music accompaniment and non-speech noise at 5–25 dB signal-to-noise ratio, the range DenoiSpeech used. We then run the mixtures through a music separator of the MDX23 family used by SingNet and through a standalone speech enhancer. Each utterance therefore exists in clean, mixed and separated form, with clean pitch, energy and voicing references available.
Diagnostic study. For each gender and each processing condition, we measure pitch-tracking error against the clean reference. We also measure the ratio of extracted pitch standard deviation to clean pitch standard deviation, localized to top-decile excursions and to onsets detected by the excitation method. To separate register from representation, we train on three gender compositions: natural LibriTTS, LibriTTS subsampled to 80% male, and LibriTTS subsampled to 80% female. We also include AISHELL-3, which is naturally female-heavy.
Repaired-target estimator. The estimator is a small convolutional network. For each frame it receives the concatenation of four inputs: log-mel features of the separated audio, the linear-prediction residual Hilbert envelope, frozen self-supervised features, and a speaker embedding. It predicts clean log-F0, voicing and energy, with its loss computed against the paired clean references. Mixtures used to train it are disjoint in speakers from the synthesis evaluation.
Synthesis and in-the-wild transfer. We train a FastSpeech-2-style synthesizer with explicit pitch, energy and duration predictors on separated data, using either the naive tracker's targets or the repaired targets. We also run a stochastic variant built on VITS with DPP sampling (DPP-TTS) to check whether the effect persists when the objective itself is not deterministic. Finally, we apply the estimator, frozen, to genuinely in-the-wild separated audio, where no clean reference exists and judgments must be perceptual.
Baselines
Naive reference. The do-nothing baseline trains the same synthesizer on separated audio with pitch from a standard tracker, run with default settings and no repair. Its gender-stratified pitch variance ratio and pitch error define the size of the problem. Paired with it is an oracle upper bound: the same synthesizer trained with clean-reference pitch targets on separated audio. This isolates how much of the loss comes from targets rather than from the audio the decoder hears.
DenoiSpeech. The strongest existing method for artifact-laden TTS data is DenoiSpeech, which conditions FastSpeech 2 on a jointly learned frame-level noise vector. It reports MOS 3.77 versus 3.66 for an enhancement-based baseline on artificial noise, and 3.35 versus 3.04 on real-world noise. We will reimplement it on our separated corpora with the same backbone, data and number of training steps, and give it the same hyperparameter search budget. Its noise conditioning and our target repair are orthogonal, so we also report the combination.
LibriTTS-R. The strongest restoration baseline is training on LibriTTS-R audio, where Non-Attentive Tacotron trained on restored train-other-500 reached MOS 4.0–4.3, against 2.4–3.4 on the original. Because Miipher is not available to rerun on our mixtures, this comparison runs on LibriTTS, where restored audio already exists.
Expressive-prosody reference. For pitch accuracy we use Mellotron, which reports an F0 frame error of 8.77% on LibriTTS against 27.5% for an E2E-Prosody baseline. We found no reported result on Lacuna for gender-stratified F0 error or pitch-variance ratio on separated speech. Our naive baseline therefore supplies that number.
Ablations. Three ablations test the key ingredient. The first removes the excitation feature. The second removes the self-supervised feature. The third replaces concatenation with cross-attention.
What counts as beating the baselines. The proposal succeeds if repaired targets close at least half of the gap between the naive baseline and the clean-target oracle in female pitch-variance ratio. It must also reduce the female-minus-male difference in that ratio without lowering MOS. Finally, it must equal or exceed DenoiSpeech MOS on the same data.
Why not just…?
Why not just restore the audio first with LibriTTS-R-style generative restoration or a strong enhancer? This is the most serious alternative. Restoration visibly raised MOS in LibriTTS-R. However, a generative restorer may repair timbre while re-synthesizing a smoothed pitch contour, and it occasionally hallucinates. Separately, DenoiSpeech found that standalone enhancement leaves distortions that decoders copy. We will measure pitch-variance ratio by gender directly on restored LibriTTS-R audio. If restoration already recovers female pitch excursions to within the oracle margin, target repair adds nothing on restorable speech. The contribution would then shrink to cases like singing and music separation, where no restorer exists, and we would say so.
Why not just drop unreliable frames, masking low-confidence pitch from the loss? Masking is cheap, and it may remove octave errors. Our hypothesis, though, is that the unreliable frames are disproportionately the expressive peaks and onsets. Discarding them would bias the predictor toward the mean, reproducing the flattening. We will run confidence masking at several thresholds as a baseline and report, per gender, the fraction of top-decile excursions it discards. If masking matches the fused estimator, we will adopt masking and drop the estimator.
Why not just rebalance speakers by gender or oversample the minority? Rebalancing addresses representation but not register-specific corruption. The AISHELL-3 condition tests this directly: female speakers are the majority there, so continued female-specific flattening under separation would show that rebalancing cannot be the whole fix. We will still run an oversampled baseline on the 80%-male LibriTTS subset.
Why not just sample more diversely, with higher temperature or DPPs? More diverse sampling spreads the output around whatever distribution the targets define. If the targets have been clipped, it spreads around a flattened distribution. DPP-TTS also notes that raising the temperature destabilizes speech. The DPP-TTS variant of our experiment tests whether repaired targets still help a model that is already diversity-seeking.
Proposed Experiments and Decisive Checks
Evaluation. Objective metrics are the pitch-variance ratio against clean references, pitch RMSE and F0 frame error, all stratified by gender, on the LibriTTS, VCTK and AISHELL-3 separated test sets. Subjective metrics are naturalness MOS and a side-by-side expressiveness preference test in the style of DPP-TTS. Raters listen to separate male and female subsets, with confidence intervals reported per subset.
First decisive check (does the problem exist?). The naive tracker's variance ratio on separated audio must be measurably lower for female than for male speakers, and this must hold on AISHELL-3 as well as LibriTTS. If the asymmetry is absent, the gender framing fails. We would then report a gender-neutral target-corruption result.
Second decisive check (is it a target problem?). The clean-target oracle must substantially outperform the naive baseline. If it does not, decoder imitation of artifacts, DenoiSpeech's mechanism, dominates, and repairing targets is the wrong lever.
Third decisive check (does fusion earn its place?). Removing the excitation feature should hurt most on separated female speech, where magnitude harmonics are most damaged. If the self-supervised feature alone does as well, the fusion story reduces to "use a pretrained feature," and we would report that plainly.
In-the-wild transfer. On singing data processed like SingNet's, which has no clean references, the test is gender-stratified listener preference for repaired-target over naive-target synthesis.
References
- Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
- CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
- Integrating Discrete Word-Level Style Variations into Non-Autoregressive Acoustic Models for Speech Synthesis
- Speaker Adaption with Intuitive Prosodic Features for Statistical Parametric Speech Synthesis
- Denoispeech: Denoising Text to Speech with Frame-Level Noise Modeling
- SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset
- LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
- Detection of vowel onset point events using excitation information
- Phase distortion statistics as a representation of the glottal source: application to the classification of voice qualities
- Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion
- Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps
- Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens
- DPP-TTS: Diversifying prosodic features of speech via determinantal point processes