Asking New Speakers the Right Sentences: Speaker-Informative Prompt Selection for Personalized Pausing

A text-to-speech system that sounds like a particular person needs to know more than their voice timbre. It also needs to know where they pause and for how long. Current systems that adapt to a new speaker's phrasing ask that speaker to record a handful of reference sentences, typically five, and those sentences are chosen at random. Some sentences, however, say almost nothing about a speaker: nearly everyone pauses at a full stop, so recording one teaches the system little. Other sentences are uninformative for the opposite reason. Pausing in them is erratic even for a single person, so one recording is mostly noise. The sentences worth recording are the ones where speakers differ systematically and each speaker is consistent with themselves.

We propose to choose those sentences on purpose. The tool is a quantity from active learning, the family of methods that let a model pick which examples to label next. The quantity is the mutual information between a sentence's pause pattern and the speaker's identity. It is high exactly where the model's pause predictions diverge across known speakers but are confident for each one. We compute it with a speaker-conditioned phrasing model, update a belief about the new speaker after each recording, and pick the next sentence to ask for. The claim to test is concrete. Under the same recording budget, measured in seconds of speech, sentences chosen this way should adapt pause location and pause duration prediction faster than random sentences, than text-only heuristics, and than plain "most uncertain" selection.

Core Claim

Uncertainty-guided selection for prosodic personalization should target uncertainty about the speaker, not uncertainty about the model's weights or raw output entropy.

Let $x$ be a candidate sentence's text and $y$ its pause labels. In $y$, each word juncture is marked as no pause, or as a brief, medium, or long pause. Let $s$ be a speaker embedding, and $q(s)$ the system's current belief about which embedding describes the new speaker. The selection score is

$$A(x) = H\big[\mathbb{E}_{q(s)}\, p(y \mid x, s)\big] - \mathbb{E}_{q(s)}\, H\big[p(y \mid x, s)\big],$$

where $H$ is entropy. The first term is the total predicted uncertainty. The second is the uncertainty that would remain even if we knew the speaker exactly, which is the speaker's own randomness. Their difference is the expected information the recording gives about $s$.

This is the standard information-gain acquisition rule, but taken over the speaker latent rather than over network weights. That choice has two consequences. First, it separates reducible from irreducible variation by construction. This is the separation the active learning literature warns is essential: Peng et al. found that maximum-uncertainty client selection pulls in noisy, unlearnable data and that median aleatoric uncertainty works better, and ACED filters by requiring a confident reference model. Second, it is cheap. In the phrasing architecture of Yang et al., the speaker embedding is injected after the text encoder. The expensive phoneme-level language model therefore runs once per sentence, and only the small decoder runs once per speaker sample. This avoids the 25–30 stochastic full passes that dropout-based estimators need, as in MCDUE and MULTIUAT.

Relation to Prior Work

Two papers motivate the gap.

Yang et al. build a speaker-conditioned phrase-break predictor on a phoneme-level BERT. They add a few-shot adapter that maps speaker-verification embeddings into the phrasing model's speaker space and works from about five reference utterances. Those references are not chosen, and the model never reports its own uncertainty. We keep their architecture and adapter fixed and change only which sentences reach it.

Yang et al.'s duration-aware pause insertion work shows two things. Speaker embeddings substantially raise respiratory-pause accuracy. Listeners tolerate small errors in pause location but are sensitive to pause duration and to mismatched speaker rhythm. This is why our labels include their three duration bins (under 300 ms, 300–700 ms, over 700 ms) and are not limited to binary breaks.

The closest prior use of active learning for speaker adaptation is two-step active learning for ASR adaptation. It decodes five sentences, builds a profile of which phones the recognizer gets wrong for that speaker, and picks sentences whose phone histogram matches that profile using a divergence measure. Our proposal differs on four counts. It targets synthesis prosody rather than recognition. It scores sentences with a model's probabilistic predictions rather than a static text histogram. It updates its belief sequentially after each recording. It explicitly discounts within-speaker randomness.

Du and Yu constrain the design. They show phone-level prosody is strongly multi-modal, but they use the mixture only to sample diverse prosody for a single speaker and never to measure confidence. Their result warns that a unimodal predictive distribution would misstate the within-speaker term in $A(x)$. We therefore keep the pause head categorical, not Gaussian.

LLM-generated phrase-break labels are speaker-agnostic and plateau perceptually. This makes them a reasonable source of the population prior, not a substitute for the speaker's own recordings.

Research Program

We first train a speaker-conditioned pause model on the LibriTTS-R training speakers (LibriTTS-R). It follows the phoneme-level encoder, speaker embedding, and decoder design of Yang et al., with a four-way head at each word juncture (no pause plus the three duration bins). Labels come from forced alignment of the audio.

The belief $q(s)$ over a new speaker is a weighted set of particles: the learned embeddings of the training speakers, of which LibriTTS-R has roughly two thousand. Before any recording, the weights are uniform. If an acoustic embedding of the speaker is available from any audio, it can supply a prior through the adapter. After the speaker records sentence $x_t$ with observed pauses $y_t$, each particle's weight is multiplied by $p(y_t \mid x_t, s)$ and renormalized.

Juncture-level scores are summed under an independence assumption, and we report how much this assumption costs. Because longer sentences trivially carry more information, every comparison charges selection by estimated recording duration, not by sentence count. We will also report the score normalized per second, so the method cannot win by picking long sentences.

After selection stops, the final adapted speaker representation is produced the same way for every method, with the Yang et al. adapter run on the selected utterances. Any difference between methods then comes from selection alone. A secondary variant uses the posterior mean embedding directly, which tests whether the belief update is itself a usable adapter.

The analysis has two strands. We will inspect what high-$A$ sentences look like: long clauses without punctuation, coordinated phrases, and parentheticals are the likely candidates. We will also check whether selection stays informative for male and female speakers alike, given the naturalness gap reported for male voices in LibriTTS.

Baselines

The naive reference point is random selection under the same duration budget, fed into the identical adapter. This is the protocol of Yang et al., who report only that the adapter "reaches competitive performance with as few as 5 reference utterances." There is no reported few-shot $F_{0.5}$ on Lacuna for that adapter, so we will reproduce it. We run random selection with many seeds so that its spread defines the noise floor.

The strongest existing reference for absolute accuracy is the same paper's model with trainable speaker embeddings learned from a speaker's full data. It reports $F_{0.5}$ of 0.4991 on LibriTTS-R, against 0.3719 for a BERT-base model without speaker conditioning. We treat these two numbers as the ceiling and the floor for break location on seen speakers. Our held-out-speaker evaluation is a different setting, so we re-measure both ends in our split rather than compare across papers.

The strongest selection competitor is a text-only coverage heuristic in the spirit of the ASR active adaptation work. It greedily picks sentences that maximize the number and syntactic variety of plausible break junctures, where "plausible" means not sentence-final and not trivially comma-marked. It runs on the same pool and budget. It is the real threat because it may capture most of what $A(x)$ finds without any model.

Two ablations isolate the key ingredient. The first, total-entropy selection, keeps only the first term of $A(x)$, the standard maximum-uncertainty rule of Song et al. and Wu et al. applied to our model. The second, speaker-agnostic entropy, uses an unconditioned pause model's entropy. A third variant computes $A(x)$ once at the start without belief updates, which separates the value of sequential querying from the value of the score itself.

All methods share the pretrained model, the adapter, its training schedule, and the hyperparameter search budget, which is tuned on validation speakers only. "Beating" a baseline has a concrete meaning: at matched budget, mean held-out-speaker $F_{0.5}$ and duration-bin macro-F1 exceed the baseline by more than the random-seed spread. The headline target is that our method at five-utterance-equivalent duration matches random selection at a substantially larger budget.

Why not just…?

Why not just use five random sentences, since the adapter already works with them? "Competitive" was established without any comparison to chosen sentences, so the claim does not tell us how much is left on the table. The ASR precedent found a modest 1.1-point gain over random at 15 sentences, which is reason for caution, not confidence. The learning-curve experiment settles this directly. If random sits within seed noise of our method at every budget, the idea is not worth deploying, and we will say so.

Why not just pick sentences with many break-eligible junctures, with no model at all? This is the alternative we take most seriously. Much of the speaker variation may simply live in long unpunctuated spans. Two pieces of evidence argue against it, though neither is decisive. MCDUE found model uncertainty beat input-space coverage on regression benchmarks. Speaker-dependent pausing tracks breathing habits rather than syntax alone (Yang et al.). The test is a head-to-head under a duration budget. We will also regress $A(x)$ on the heuristic's features and report what fraction of selection overlap and gain the heuristic explains. If it explains nearly all of it, the contribution reduces to a validated heuristic.

Why not just query where the model is most uncertain? Total entropy rewards sentences where a speaker's own pausing is random, so recording them mostly adds noise to the belief update. This is the trap documented by Peng et al. and implied by the multi-modality result of Du and Yu. The first-term-only ablation measures this cost. We also report the within-speaker entropy of the sentences each method selects.

Why not just generate speaker data with an LLM? LLM phrase-break labels are text-oriented and carry no speaker information, and their perceptual benefit plateaus. They can improve the population model, and we will try them as pretraining augmentation. They cannot reveal one person's habits, which is the quantity being estimated.

Proposed Experiments

Simulation setup. The main experiment is a pool-based simulation on LibriTTS-R. Held-out speakers with large numbers of utterances each contribute a candidate pool and a disjoint test set drawn from their own recordings. Because audiobook speakers read different texts, each speaker's pool is limited to sentences that speaker actually recorded. This is a known limitation of simulation. A deployed system would query from a shared prompt bank, which motivates a small follow-up recording study.

Metrics and curves. We trace learning curves of pause-location $F_{0.5}$ and duration-bin macro-F1 on each speaker's test sentences as the budget grows from one to twenty utterance-equivalents. Prominence labels in the style of Talman et al. serve as a secondary target, to see whether pause-informative sentences also inform emphasis.

Robustness check. We corrupt a fraction of pool labels with shuffled pause durations. We then check whether the sequential belief update degrades gracefully, and whether total-entropy selection is drawn toward the corrupted items more than $A(x)$ is.

Perceptual test. For a subset of speakers, we synthesize test passages with Matcha-TTS, the vocoder Yang et al. used to report MOS of 3.35 to 3.51. Listeners judge in AB tests which rhythm better matches the speaker's real recordings, comparing pauses predicted after random versus actively selected adaptation at equal budget. Given the duration sensitivity reported by Yang et al., this test is the decisive one.

What would kill or narrow the idea. The idea fails if, at matched recording duration, $A(x)$ does not beat both random and the juncture-coverage heuristic beyond seed noise on duration-bin F1, or if listeners cannot hear the difference. It narrows to "subtracting within-speaker randomness is what matters" if the static score matches the sequential one.