Pitch-Blind Voice Adaptation: Adapting Only the Vocal-Tract Path So Few-Shot Voices Survive Expressive Prosody

When a text-to-speech system is personalized to a new speaker from a minute of audio, that audio is almost always calm, read-style speech. Users then want the cloned voice to sound angry, surprised, or excited. Such speech has pitch ranges and rhythms the adaptation data never contained. We propose to adapt only the part of a parallel speech generator that models the vocal tract, and to structure the model so that this part never sees pitch. The adapted voice identity should then carry over to expressive pitch contours instead of breaking or drifting when the prosody moves outside what the few adaptation sentences covered. The test is concrete. Adapt on a handful of a speaker's neutral utterances, then synthesize that same speaker's emotional utterances, and compare against their real emotional recordings.

This matters because the two strongest threads in the literature point in opposite directions. Parameter-efficient adaptation methods confine their updates to the spectrogram decoder. Expressive methods need rich control over pitch and duration. Nobody has checked whether a decoder-side adapter fitted on neutral speech actually generalizes when the pitch input to that decoder is pushed into an emotional range. A non-autoregressive model, meaning one that generates all spectrogram frames in parallel rather than one after another, gives us the explicit pitch and duration inputs needed to run exactly this test.

Core Claim

Existing adapters for parallel text-to-speech modulate a decoder that mixes phoneme and pitch information in the same hidden layers. AdaSpeech conditions layer-normalization scales and biases on the speaker. Residual Adapters for Few-Shot TTS inserts small bottleneck layers into a frozen Conformer decoder. In both cases the adapter can learn a speaker transform that is only valid on the narrow pitch range of the adaptation data.

Our claim is that timbre adaptation extrapolates to unseen expressive pitch only when the adapted parameters sit in a pathway that is structurally independent of pitch. FastPitchFormant supplies that structure. It splits generation into a formant generator, which sees only text and models the vocal-tract filter, and an excitation generator, which sees text and pitch and models the glottal source. That paper showed the split preserves identity under large pitch shifts, but only for a single Korean speaker with no adaptation.

We extend the split to a multi-speaker English model and restrict few-shot adaptation to three things. The first is conditional layer-norm parameters in the formant generator. The second is an optional handful of frequency-warp parameters, in the spirit of the 80-parameter filter adaptation in SincNet adaptation, where tuning filter cut-offs alone produced an emergent vocal-tract-length warp for children's speech in speech recognition. The third is a two-number per-speaker affine on log-pitch, a mean and a range, that re-centers the predicted pitch contour for the new speaker.

The pitch and duration predictors stay frozen. This respects the negative result in the Residual Adapters paper, which found that adapting the variance predictors degraded quality.

Research Program

Base model. We first build a multi-speaker version of the source-filter model on LibriTTS. It has a shared text encoder, frozen-at-adaptation duration and pitch predictors, a pitch-blind formant generator carrying speaker-conditioned layer normalization, and an excitation generator that receives text and pitch. The excitation generator gets either no speaker conditioning or, in an ablation, a tiny speaker adapter of its own.

Adaptation. For each new speaker we tune roughly five thousand parameters on 1, 5, or 20 neutral sentences. This matches the AdaSpeech footprint, so any gain cannot come from extra capacity.

Measuring whether the split holds. We synthesize fixed text under the adapted voice while sweeping the pitch input across and beyond the adaptation range. We then track speaker-verification similarity and spectral-envelope stability. The prediction is that the pitch-blind adapted voice shows a flat similarity curve. A decoder adapter should instead lose similarity as pitch leaves the adaptation range.

Known physical limitation. In real speech, formants do shift somewhat with pitch and vocal effort. A strictly pitch-blind filter cannot model this. We therefore measure the residual spectral error on real emotional recordings rather than assume it is negligible. The excitation-adapter ablation tells us how much of a speaker's identity actually lives in the source rather than the filter.

Baselines

Naive reference. The do-nothing reference is the unadapted multi-speaker base model, conditioned only on a speaker embedding extracted from the adaptation audio. Tuning only such an embedding gave a similarity score of 3.34 on VCTK in AdaSpeech, which sets the floor.

Standard parallel backbone with adapters. The main competitor is a FastSpeech 2–style model trained on the same LibriTTS data, with the same parameter budget spent in two alternative ways. The first is AdaSpeech conditional layer normalization across the whole decoder. The second is residual adapters in the decoder, following Residual Adapters for Few-Shot TTS. AdaSpeech reports a similarity mean opinion score of 3.96 on VCTK. Residual adapters report a naturalness score of 4.46 against 4.49 for full fine-tuning, on a different backbone. We reimplement both rather than compare to those numbers directly.

Strongest expressive system. GenerSpeech reports naturalness 4.06 and speaker cosine similarity 0.88 on unseen VCTK speakers, and naturalness 4.11 on unseen ESD emotions, with 1–20 adaptation samples. It is the strongest existing system that handles both style and few-shot speakers, so we run it with its published configuration on the same splits.

Structural ablation. The key ablation keeps our exact architecture and budget but places the adapter in the excitation generator, or in both generators, instead of the pitch-blind formant path. This isolates whether pitch-independence of the adapted pathway is what matters.

Fairness and the bar to beat. All systems use the same pre-training data, the same vocoder, the same adaptation sentences, and the same number of adaptation steps tuned on a shared validation speaker. Beating the baselines means two things on the emotional evaluation described below. Speaker similarity must be significantly higher at matched naturalness. The similarity gap over the decoder-adapter baselines must widen as the emotional pitch range grows. That widening pattern is the claim's signature, and a uniform offset would not support it.

Why not just…?

Why not just use AdaSpeech-style decoder adaptation and feed it expressive pitch? This is precisely the untested assumption. Nothing in AdaSpeech or Residual Adapters evaluates adapted voices under prosody outside the adaptation range. Their decoders see pitch jointly with text, so the adapter is free to fit pitch-specific spectra. The pitch-sweep curves and the emotional evaluation directly show whether this breaks. If it does not break, our claim fails.

Why not just pitch-shift the adaptation audio to cover a wider range and train ordinary adapters? This is the most serious threat, because data augmentation is cheaper than an architectural change. A naive pitch shift also moves formants, which teaches the wrong timbre. A formant-preserving shift, done by analysis-resynthesis that keeps the spectral envelope fixed, does not have that problem, but its artifacts may be memorized by the adapter. We include formant-preserving augmentation of the adaptation set plus decoder adapters as a full baseline. If it matches our method on emotional similarity, we will report that the gain comes from pitch coverage rather than architecture. We will then recommend augmentation, since it is simpler.

Why not just run a post-processing voice converter on an expressive base voice? Voice Filter maps the output of a frozen text-to-speech system to a target timbre, conditioned on pitch. In principle it could re-voice emotional synthesis. Its converter is trained on synthetic parallel data from a neutral engine, though, and faces the same pitch-range extrapolation question. It also cannot carry the speaker's own pitch register or rhythm. We run it on top of the same expressive base model and score it identically.

Why not just fine-tune everything? The file's evidence argues against this with few sentences. Continual Speaker Adaptation and Voice Filter report overfitting and forgetting in this regime, and full fine-tuning costs about 1,000 times more storage per thousand speakers than adapters. We still include full fine-tuning at 20 sentences as an upper reference.

Proposed Experiments

Decisive experiment. The main experiment uses the English speakers of ESD, which has 350 parallel utterances per speaker in neutral, happy, angry, sad, and surprise. We pre-train on LibriTTS, hold out all ESD speakers, and adapt each ESD speaker on 1, 5, or 20 neutral utterances. We then synthesize held-out emotional sentences in two conditions.

In the first condition, the true duration and pitch contours are taken from that speaker's real emotional recording. This isolates timbre under expressive pitch from any failure of prosody prediction. With the true timing, we can compute mel-cepstral distortion frame-aligned against the real recording, alongside speaker cosine similarity and a similarity opinion score. In the second condition, prosody is transferred from a different ESD speaker's emotional recording, which tests realistic use.

Metrics and reference points. Emotion is checked with a speech emotion classifier. The ESD paper reports 89% classification accuracy on English, which serves as the reference for real speech. Intelligibility is measured with ASR word error rate. For cross-speaker transfer, we additionally report pitch correlation against the reference contour, where Daft-Exprt reports 0.43. We also report naturalness opinion scores. No reported result was found on Lacuna for neutral-adapted voices scored against the same speaker's real emotional recordings, so this benchmark protocol is itself a contribution.

Secondary checks. On VCTK with 1–20 sentences, we confirm that the pitch-blind restriction costs little on ordinary read speech relative to AdaSpeech and GenerSpeech. We also measure speed to confirm the model remains single-pass parallel generation.

What would refute the claim. The claim is refuted if any of three things happens on emotional ESD speech at a matched parameter budget. Decoder adapters could match our similarity. The formant-preserving augmentation baseline could close the gap. The excitation-path ablation could perform as well as the formant-path adapter.