Text-to-speech systems driven by prompts can now produce a voice from a sentence such as "a deep, calm male voice." They treat that sentence as a single fixed point, though. A sound designer who wants a voice "70% of the way from timid to authoritative," or a blend of two described speakers that no single word captures, has no reliable control. Speech research has long shown that acoustic speaker spaces can be traversed smoothly by mixing two speaker vectors. Classification research has shown that replacing arbitrary class indices with natural-language label descriptions makes models generalize better. No one has asked whether the space of voice descriptions can be traversed the same way.
This note proposes training a prompt-conditioned speech synthesizer so that mixing two description embeddings by a fraction $\alpha$ moves the perceived voice attribute by the same fraction $\alpha$, and leaves unrelated attributes alone. The supervision comes from something existing corpora already contain: graded labels. The LibriTTS-P corpus describes pitch, speed and loudness at five quantized levels and rates speaker impression words by intensity. Each intermediate level is therefore a ground-truth point that an interpolation between the extreme levels should hit. Smoothness alone is weak, because a path can be smooth without being predictable. The new property is calibration: the position along the path and the position in perception agree, and this is tested on levels and word blends held out from training.
Core Claim
Interpolation in a semantic description space can be made perceptually calibrated by anchoring it to graded descriptions. Generic interpolation regularizers do not achieve this.
A prompt encoder is a model that turns a description into a vector. Let $e(d)$ be the vector it produces for description $d$. Take two descriptions $d_i$ and $d_k$ of an attribute at ordinal levels $i<k$, and an utterance $x_j$ labeled at an intermediate level $j$. We require that the mixed embedding at $\alpha_j = (j-i)/(k-i)$ produce speech whose measured attribute matches $x_j$:
$$\mathcal{L}_{\text{anchor}} = \big\| a\big(G(\alpha_j e(d_k) + (1-\alpha_j) e(d_i))\big) - a(x_j) \big\|^2 .$$
Here $G$ is the synthesizer and $a(\cdot)$ is a differentiable attribute predictor, such as an $F_0$, speaking-rate, loudness or impression-intensity estimator. Off-axis attributes enter the same loss with targets held fixed.
The claim is falsifiable in two ways. If naive interpolation of an off-the-shelf prompt encoder is already calibrated, the anchor loss is unnecessary. If writing intermediate words ("moderately high pitch") is just as calibrated, embedding interpolation is unnecessary.
Relation to Prior Work
The prompt-to-speech line supplies the backbone and the data, and it motivates the gap. PromptTTS conditions synthesis on a style vector from a BERT encoder trained to predict discrete attribute buckets. InstructTTS aligns style prompts to speech through cross-modal metric learning, but it generates discrete codebook tokens and deliberately removes speaker identity from the prompt. PromptTTS 2 fills in what a description leaves unspecified by sampling Gaussian noise in a diffusion "variation network." LibriTTS-P adds human speaker-identity prompts with impression intensities and benchmarks PromptTTS++. All four map a description to a static point. None tests what happens between points, and none uses graded labels as interpolation targets.
The acoustic interpolation line supplies a method, which we reuse in a new space. Nakai et al. mix two acoustic speaker vectors and train a discriminator that must recover the mixing coefficient, which keeps intermediate voices natural. kNN-TTS blends retrieved self-supervised speech frames, and ELF blends codebook features. These methods interpolate only between voices someone has recorded. They also optimize naturalness along the path, not agreement between $\alpha$ and perception.
The label-description line supplies the motivating principle. Song et al. classify by proximity to label concept vectors. Label-description training fine-tunes on curated descriptions, reporting +17.5/+18.9 point average gains and less sensitivity to prompt wording. Chai et al. learn descriptions that cut BeerAdvocate error from 27.8% to 15.6%. These results suggest that descriptions carry ordinal and relational structure. Our anchor loss uses that structure directly, exploiting graded descriptions the way label-description training exploits definitions.
Two papers constrain the design. VECL-TTS finds that pushing emotional style lowers identity similarity (DMOS 3.95 to 3.81). PromptTTS 2 shows that text under-specifies timbre. We respond to the first by measuring off-axis drift explicitly. We respond to the second by restricting the claim to attributes the descriptions actually name, and by keeping a fixed residual timbre code along each path.
Research Program
The synthesizer is a FastSpeech 2–style non-autoregressive model, matching the backbone of Nakai et al. so their critic transfers directly, with a BigVGAN vocoder. The prompt encoder is a pre-trained text encoder whose output is split into an identity part, fed by LibriTTS-P speaker prompts, and a style part, fed by style prompts. This split takes a position in the InstructTTS-versus-LibriTTS-P disagreement: identity belongs in the description, but in its own subspace. A low-dimensional residual code, inferred from reference audio during training and held constant along any path at test time, absorbs timbre the text cannot specify.
Training combines three losses. The first is standard reconstruction. The second is the anchor loss above, applied to quantized pitch, speed and loudness levels and to impression-word intensity ratings. Attribute predictors are trained on LibriTTS-P and validated against human ratings before use. The third is a Nakai-style critic applied to mixtures of description embeddings, including combinations with no labeled intermediate, so that naturalness holds where anchors are absent.
The main analysis is generalization of calibration. Anchors are trained only on levels 1, 3 and 5 and on a subset of impression-word pairs. Paths are then evaluated at the held-out levels 2 and 4 and on unseen impression-word blends.
Baselines
The naive reference point is linear interpolation of description embeddings in a prompt TTS model trained on LibriTTS-P with no interpolation-specific loss: a PromptTTS++-style system as benchmarked in LibriTTS-P. That paper reports a naturalness MOS of 3.37 for PromptTTS++ on LibriTTS-P. We retrain it with our backbone, vocoder and data so that only the losses differ. To beat it, our model must show higher rank correlation between $\alpha$ and the measured attribute, lower error at held-out levels, and no MOS loss at the endpoints.
The strongest existing interpolation method is the multi-task interpolation critic of Nakai et al.. It reports MOS 3.50 for unseen speakers on JVS, against 3.13 for FastSpeech 2. We run it in two forms with the same tuning budget. In the first, the critic is applied to our description embeddings without the anchor loss. In the second, a text-to-speaker-vector predictor is followed by the critic in the acoustic space. Beating it means matching its path naturalness while being significantly better calibrated at held-out levels.
For the stochastic route we compare against PromptTTS 2, which reports MOS 3.88 and speech similarity 0.355 on Multilingual LibriSpeech. It provides no interpolation control, so we interpolate its prompt representation with a fixed noise seed. No reported result was found on Lacuna for interpolation monotonicity or calibration for any system, so all path metrics will be produced fresh under identical conditions.
The ablations remove one ingredient at a time: the anchor loss, the critic, the identity/style split, and the fixed residual timbre code. The anchor-loss ablation is the one that isolates the central claim.
Why not just…?
Why not just interpolate the prompt embeddings of an existing model? It may already work. Pre-trained text encoders carry some ordinal structure, as the label-description results suggest. This is the naive baseline. If its held-out-level error matches ours, the anchor loss is unjustified and we will report that.
Why not just write intermediate words like "slightly" or "moderately"? This is the most serious threat. LibriTTS-P's templates already express five levels in words, so for pitch, speed and loudness, LLM-written intermediate prompts could be as well calibrated as embedding mixes. We test this directly by generating intermediate-word prompts for the held-out levels. Our expected advantage lies where the lexicon runs out: continuous values between named levels, and blends of two impression-word identities such as "cute" and "cool," which have no compositional phrase. If words win even there, the contribution narrows to a training recipe that improves calibration of worded prompts.
Why not map text to an acoustic speaker vector and interpolate there with Nakai's critic? The acoustic space is denser and already regularized. However, the critic rewards naturalness and recoverability of $\alpha$, not agreement between $\alpha$ and perception. A text-to-acoustic predictor also gives no axis-specific control, so moving pitch may drag identity, which is the trade-off VECL-TTS observed. The second Nakai baseline tests this alternative, and off-axis drift is measured for every system.
Why not just sample diffusion noise? Noise sampling gives diversity, not a requested position. A fixed-seed interpolation of PromptTTS 2 is included to show whether its prompt space happens to be calibrated.
Proposed Experiments
Training and in-domain evaluation use LibriTTS-R audio with LibriTTS-P prompts. The PromptSpeech real subset serves as an out-of-corpus check, and VCTK tests generalization to new speakers.
For each attribute path we measure the objective attribute along nine values of $\alpha$. From these we compute Spearman correlation with $\alpha$, $R^2$ of a linear fit, and absolute error at held-out levels. Off-axis drift is the change in non-target attributes and in speaker-embedding cosine similarity between path endpoints, so that changing pitch does not change identity. Naturalness MOS is collected at $\alpha \in \{0, 0.25, 0.5, 0.75, 1\}$, with SPAM (SPAM) as an automatic adherence check at the endpoints. Human listeners rate impression intensity on blended identity paths, and these ratings validate the automatic intensity predictor.
The decisive check is held-out-level calibration. The proposal succeeds if anchoring only on levels 1, 3 and 5 lowers error at levels 2 and 4 relative to both the naive baseline and intermediate-word prompts, and does so without endpoint MOS falling below the naive baseline. The same must hold for impression-word blends never seen as anchor pairs. The proposal fails if intermediate-word prompts match it on blends, or if the critic alone matches it on calibration. Either outcome would show that the gain comes from vocabulary or regularization rather than from graded semantic anchoring.
References
- Prompttts: Controllable Text-To-Speech With Text Descriptions
- InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt
- PromptTTS 2: Describing and Generating Voices with Text Prompt
- LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
- Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech
- kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speech
- ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesis
- On Dataless Hierarchical Text Classification
- The Benefits of Label-Description Training for Zero-Shot Text Classification
- Description Based Text Classification with Reinforcement Learning
- VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speech
- SPAM: Style Prompt Adherence Metric for Prompt-based TTS