Phrasing Under a Time Budget: Joint Duration–Pitch Planning for Length-Aware Speech Synthesis

Human speakers plan the melody of a sentence before they say it. A sentence that will be long starts on a higher pitch and falls slowly. A short one starts lower and falls steeply. This gradual fall, called declination, is not passive air loss from the lungs. Its slope is tuned to the time the speaker expects to need (Yuan et al., broadcast news; De Looze et al., conversation). Modern text-to-speech systems do not plan this way. They predict how long each sound lasts and what pitch it carries in separate, one-way modules (FastSpeech 2, FastPitch). The one system that accepts a target total duration as input reallocates timing well but predicts no pitch at all (Total-Duration-Aware Duration Modeling).

We argue that the hard part is not making pitch aware of total length. That is a single extra input. The hard part is deciding where the pitch resets. A long utterance under a fixed time budget is usually split into several shorter stretches separated by pauses. Declination restarts in each stretch, so the slope tracks the length of the stretch, not of the whole sentence. Each break is a pitch event (a reset) and a timing event (a pause plus lengthening of the preceding syllable) at the same moment. Under a fixed budget, every break also takes time from somewhere else. We propose a synthesizer in which a timing model and a pitch-planning model refine each other's tentative decisions over a few rounds under a shared time budget. We test whether this exchange is needed at all, compared with simply conditioning pitch on length.

Core Claim

The length–declination relationship found in human corpora holds within the stretches of speech between pauses. Yuan et al. found it in utterances already split at pauses longer than 50 ms. De Looze et al. found slope scaling both for stretches between silences and for whole turns, but found higher starting pitch for longer units only at the turn level. Where those stretches begin and end is a joint timing-and-pitch decision. Our claim is that a model can reproduce human length-scaling of declination in multi-phrase speech under a time budget only if timing and pitch decisions can revise each other.

A single forward pass that conditions pitch on total or predicted length should do nearly as well on single-phrase utterances. It should fall measurably short when pauses must be placed and traded off against the budget. That split is the falsifiable prediction. If global length conditioning matches the joint model everywhere, the mutual-refinement half of the idea is wrong, and we will report that.

Relation to Prior Work

The phonetics papers motivate the target behaviour. Yuan et al. show the slope–length correlation across 5,652 English and 8,383 Mandarin utterances, and it survives removal of the first and last 500 ms. De Looze et al. show it in spontaneous Irish English dialogue and add the higher-starting-pitch effect for longer turns. O'Reilly et al. extend it to Irish declaratives and questions.

The case that pitch also drives timing, not just the reverse, comes from French focus. In Dohen et al., signalling focus lengthens focal syllables by about 34% and the preceding syllable by 20%, and it re-phrases the preceding material. Cui et al. tie duration to final pitch slope in Northeastern Mandarin. Pierrehumbert's account of Japanese intonation (Pierrehumbert et al.) supplies the structural idea we reuse: focus inserts a phrase boundary that resets the step-down in pitch, so phrasing determines the pitch domains.

Two papers constrain how much to promise. Gendrot et al. find late pitch upswings in spontaneous speech. Clean planned declination is therefore a property of read speech, and we restrict claims to read speech. Cheng and Gubian predict tone contours from syllable duration, but only for isolated two-syllable nonsense words. That is evidence for a duration-to-pitch link, not a sentence-level model.

On the synthesis side, three systems set the baselines. FastSpeech 2 and FastPitch use independent or one-way-cascaded duration and pitch predictors with no notion of total length. The total-duration-aware model (Eskimez et al.) supplies our timing method. It is a masked bidirectional transformer that treats phoneme durations as discrete tokens and fills them in iteratively, given the target total duration. We extend it with a pitch planner that it lacks. NaturalSpeech learns duration and pitch jointly but implicitly, inside a latent space, without an explicit budget or an interpretable declination.

The refinement mechanism comes from vision and natural-language work, where two coupled predictions repeatedly read each other's tentative outputs. The Recurrent Interaction Network feeds entity and relation probabilities back and forth and loses 8–10 exact-match F1 points without the exchange. MuLA finds that one-way adaptation hurts the source task while two-way adaptation helps both, and that gains saturate after about five rounds. None of these touches speech. We borrow the loop and test whether MuLA's asymmetry holds for timing and pitch: does feeding pitch plans back improve duration quality itself?

Model

The pitch planner outputs a compact, interpretable plan rather than a frame-level contour. It decides a break probability at each word boundary. For each stretch $j$ between breaks, it outputs a starting pitch $p_j$, a slope $s_j$, and a final-lowering amount. A small residual network then adds local pitch accents. The stretch-level baseline is $F_0(t) \approx p_j + s_j\,t$ for $t$ within stretch $j$ of length $T_j$. This is the same linear-regression description Yuan et al. use to measure declination, so model and evaluation share a definition.

Given text $x$ and a time budget $B$, rounds $k = 1,\dots,K$ alternate two updates:

$$d^{(k)} = \mathrm{Dur}\big(x, B, \pi^{(k-1)}\big), \qquad \pi^{(k)} = \mathrm{Pitch}\big(x, B, d^{(k)}, \{T_j(d^{(k)})\}\big)$$

Here $d^{(k)}$ is the duration tokens after round $k$ and $\pi^{(k)}$ the pitch plan. $\{T_j(d^{(k)})\}$ are the stretch lengths implied by the current durations and break decisions, and $\pi^{(0)}$ is a neutral plan.

The timing model is the masked decoder of Eskimez et al. At each round it re-masks the duration tokens near positions where the break probability changed, plus its least-confident tokens. The pitch planner sees continuous stretch lengths. This handles the mix of discrete timing tokens and continuous pitch parameters by exchanging soft break probabilities and stretch lengths rather than raw tokens. Following MuLA's saturation finding we use $K \le 4$, and we detach gradients between rounds beyond the second to avoid unrolling instability.

The total-duration constraint is enforced on the final durations as in the original timing model. Every break the pitch planner proposes must therefore be paid for by compressing other material. The acoustic model and vocoder (BigVGAN) are shared and frozen across all compared systems, so differences come only from the timing and pitch plan.

Research Program and Data

The main training set is LibriTTS: 585 hours, 2,456 speakers, split at sentence boundaries, which keeps sentence-level melody intact. Durations come from forced alignment, and stretches are defined as Yuan et al. define them, as units separated by pauses longer than 50 ms. Evaluation follows the LibriSpeech protocol of the total-duration-aware paper. During training, the budget is each utterance's true duration, randomly perturbed by up to 2x as in that work. For stretch tests at fixed text, we sweep the budget from 0.7x to 2x.

The phonetic reference distributions are measured with identical scripts on held-out LibriTTS read speech and on the 1997 English Broadcast News corpus (LDC98S71). The Mandarin Broadcast News corpus (LDC98S73) is a secondary reference, used to check whether a Mandarin-trained version reproduces the reported pattern of falling peaks with flat valleys. We treat this only as an extension.

The De Looze conversational data is ten minutes from two speakers. We use it only as a qualitative check, never for training. We make no claims about spontaneous speech, given Gendrot et al.'s upswing finding.

Baselines

The naive reference point is FastPitch trained on LibriTTS, with no budget input. It meets the budget by linearly stretching or compressing its predicted durations, with its pitch contour carried along unchanged. This is what deployed systems do when asked to fit a time slot. Eskimez et al. report that linear scaling of this kind reaches 15.1% word error rate at 2x speed, against 7.4% with total-duration-aware timing. Any proposed system must clearly beat it on intelligibility and on declination fidelity.

The strongest existing competitor is a cascade. Durations come from Eskimez et al.'s masked total-duration-aware model, which reports a Fréchet Duration Distance (a measure of how far the distribution of generated durations is from natural ones) of 0.206, against 0.403 for regression and 0.318 for flow matching. Pitch comes from a FastPitch-style predictor reading the expanded states. Its intelligibility is the 7.4% at 2x quoted above. No declination-fidelity result for it or any synthesizer was found on Lacuna, so we will measure one. It is strongest because its timing is the best reported and it already respects the budget; its only gap is pitch.

The key ablation, and the competitor we expect to be hardest to beat, is the same cascade with the pitch planner given the budget $B$ and the stretch lengths implied by the cascade's own durations, in one forward pass with no feedback. This isolates the claim exactly. If it matches the joint model, global and stretch-level length conditioning is enough and the refinement loop adds nothing.

A second control unrolls the same number of rounds and parameters, but each module only re-reads its own previous output, never the other's. This separates gains from extra computation from gains from cross-task exchange, the same distinction the Recurrent Interaction Network's ablation drew.

All systems share the acoustic model, vocoder, training data, alignments, and number of training steps. Each gets the same hyperparameter search budget (twelve configurations) and three seeds.

Beating a baseline means a difference larger than the spread across seeds, decided in advance. The difference must appear on the multi-stretch subset's slope–length agreement and break placement. Word error rate and duration distance must be no worse than the strongest cascade's own measured values in our runs.

Why not just…?

Why not just give the pitch predictor the total duration as an extra input? This is the most serious alternative. For single-stretch utterances, the 1–4 s range Yuan et al. analysed, we expect it to do nearly as well as the joint model, and we will say so if it does. Its weakness is structural. Total length is the wrong variable once speech is broken by pauses, because declination restarts at each break and the slope tracks stretch length. The one-pass ablation that also receives the cascade's stretch lengths goes further and fixes the breaks before pitch is considered. Under a tight budget it cannot move a break to fit a reset or a focus-driven lengthening, the coupling Dohen et al. document. The multi-stretch subset and the budget sweep are designed so this alternative and the joint model make different predictions.

Why not just apply a fitted declination rule after synthesis? One could fit the slope–length regression from broadcast news and rewrite FastPitch's contour stretch by stretch. This tests whether learning is needed at all, so we include it as an additional post-processing baseline on top of the strongest cascade. It cannot place breaks, and it imposes a population-average slope on every speaker and sentence type. We expect it to win on the regression metric it was fit to, but lose on listener preference and on agreement for individual utterances. If it also wins with listeners, the learned planner is unnecessary for read English.

Why not just let a joint latent model learn everything implicitly? NaturalSpeech reaches a comparative opinion score of -0.01 against human recordings on LJSpeech, so implicit joint modelling clearly yields natural speech. It has no budget input, though, and its declination cannot be read out or controlled. The fair version of this alternative within our compute is the equal-compute unrolled control, a deeper shared network without explicit exchange. If that control matches the joint model, "mutual refinement" amounts to adding capacity.

Proposed Experiments

The decisive measurement is how well the model reproduces human slope–length scaling. For every test utterance, synthesized at its true total duration, we compute, per stretch, the linear-regression pitch slope (excluding the first and last 500 ms where length allows) and the initial peak. We then regress slope on stretch length and peak on utterance length. The score is the gap between the synthetic and natural regression coefficients, plus the correlation between synthetic and natural slopes for individual utterances. We report this separately for single-stretch and multi-stretch utterances.

The prediction is that every system with length conditioning approaches natural values on single-stretch speech. Only the joint model should close most of the remaining gap on multi-stretch speech. We also score whether predicted pauses land at the reference break positions, using an F1 score.

The second check is the budget sweep. Each test sentence is synthesized at budgets from 0.7x to 2x. We measure ASR word error rate (to replicate the total-duration-aware comparison), duration distance, the number of breaks as a function of budget, and how slopes change. We expect fewer breaks and steeper slopes at tight budgets. Natural corpora establish the relationship only across different utterances, not within one text spoken at different rates; no such evidence is in the file. For this sweep we therefore rely on listeners. Paired forced-choice tests compare the joint model with the one-pass ablation at matched budgets, alongside a standard opinion-score test at natural budgets.

The third check tests whether MuLA's asymmetry transfers to speech. If the duration distance of the joint model's own durations improves over the cascade's, pitch feedback helps timing as two-way adaptation helped pose and parsing. If it worsens, the coupling costs timing quality, which would argue for a one-way design with explicit stretch conditioning.

Finally, we vary the number of rounds from 1 to 4 to see whether gains saturate early as MuLA found, and report the added inference latency against FastPitch.