When a text-to-speech system reads a novel aloud, the pitch and rhythm of each sentence depend on the sentences around it. A question that sounds neutral in isolation may need a rising contour because of the reply that follows. A calm line may need tension because a climax is two sentences away. Context-aware synthesizers exploit this by feeding the model neighboring text. The literature, however, keeps finding the same awkward fact: more context helps only up to a point. On a Japanese audiobook corpus, 64 characters of surrounding text beat 128 (Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts). On Mandarin fiction, a five-sentence window beat nine (MSStyleTTS). Both results are averages over a test set. They say nothing about whether the best amount of context is the same for every sentence.
This proposal bets that it is not. We train a small committee of prosody predictors, identical except for how far each looks into the surrounding text. A lightweight weighting network then decides, sentence by sentence, how much to trust each one. The committee's pitch, energy and duration predictions are blended before the waveform decoder. The weighting network sees only the full text and the speech already produced, never future audio. It is trained on held-out data so that it learns which horizon generalizes, not which one memorized the training chapters. The first experiment checks whether an oracle that picks the best horizon per sentence beats the best fixed horizon by a margin worth chasing. If it does not, the project stops there.
Core Claim
The claim is that the optimal context horizon for prosody is sentence-specific, and that a weighted ensemble over horizon-specialized experts can recover most of the per-sentence oracle gain. The weights must be conditioned on the input, fit on held-out data, and smoothed across sentences. Neither a single fixed-window model nor a uniform average of experts should be able to match it.
Concretely, let $p^{(m)}_t$ be the phoneme-level prosody prediction (log-$F_0$, energy, log-duration) of expert $m \in \{1,\dots,M\}$ at phoneme $t$. Each expert reads $k_m$ sentences of text on each side. The blended prediction is
$$\hat{p}_t = \sum_{m=1}^{M} w_m(s)\, p^{(m)}_t, \qquad w(s) = \mathrm{softmax}\big(g(z_s, a_{<s})\big),$$
where $s$ indexes the sentence containing phoneme $t$, $z_s$ is a bidirectional text encoding of that sentence and its neighbors, $a_{<s}$ is the acoustic style history of already-synthesized sentences, and $g$ is a small network. To avoid the jarring pitch resets and emotional jumps flagged as an open problem in long-form synthesis, the logits of $g$ are smoothed across consecutive sentences with a learned exponential moving average. This also discourages the collapse onto a single expert that greedy online weighting is prone to.
Relation to Prior Work
The closest prior work is Weighted neural network ensemble models for speech prosody control. It combined feed-forward and recurrent prosody predictors with performance-based exponential weights. It reported a 24% improvement in $F_0$ prediction over the best single recurrent model and found weighted averaging better than plain averaging. That paper supplies our basic method and motivates it. It differs from this proposal in three ways. Its weights are static, derived from aggregate training performance. Its members differ only by bootstrap resampling, not by what context they see. And it works with local phonetic features on about 30 minutes of speech, with no discourse context.
Two audiobook systems handle context inside a single model, and they constrain our design:
- Context-aware Coherent Speaking Style Prediction with Hierarchical Transformers uses a hand-designed mixture attention mask: bidirectional over text within a fixed $2N+1$-sentence window, causal over preceding speech style.
- ATCE adds a preceding-acoustic embedding and a bilateral-text embedding by vector summation.
Both accept that future audio cannot be observed. Our weighting network obeys the same rule: bidirectional in text, causal in audio. Both also fix the horizon for all sentences, and that is the assumption we test.
M2CI-Dubber is the strongest contrary signal. Its gated context-aware adaptor already weights preceding against succeeding context inside one network. If an in-network gate suffices, separate experts are unnecessary. We therefore run this design as a head-to-head baseline rather than dismissing it.
On the ensembling side, two papers motivate why single-model selection is fragile and why averaging reduces instance-level arbitrariness:
- The NN5 study (A review and comparison of strategies for multi-step ahead time series forecasting) found that combining forecasts beats validation-based winner-take-all selection.
- Individual Arbitrariness and Group Fairness shows that equally accurate models disagree on individual inputs, and that convex combinations suppress this disagreement exponentially in the number of members.
Both results raise a threat we must separate out. Some of any gain could come purely from averaging several seeds, not from horizon diversity.
Bidirectional sequence modeling enters as an efficiency constraint rather than the core mechanism. Running $M$ context encoders would be costly with quadratic attention over long windows. We instead build every expert's sentence-level context encoder as a bidirectional linear recurrent model in the style of BLUR. BLUR computes forward and backward states with parallel scans in linear time and merges them with a gated linear unit. It reports large speedups over Transformer and state-space baselines on time-series tasks. BLUR has never been applied to speech context, so we treat its efficiency in this setting as something to measure, not assume.
Research Program
The system starts from a FastSpeech 2–style acoustic model with its phoneme encoder, decoder and vocoder shared and frozen after pretraining. Each expert adds its own context encoder and variance predictor.
Four experts see 0, 1, 2 and 4 sentences of text on each side. The widest deliberately exceeds the five-sentence window reported as optimal, so the ensemble must learn when wide context hurts. A fifth expert sees only preceding synthesized acoustic style plus the current sentence. It stands in for the "causal audio" side of the asymmetry.
The weighting network is trained by stacking: we split the training chapters, train the experts on one part, and fit the weighting network on predictions for the other part. We then retrain the experts on everything, with the weighting network frozen.
The analysis program has three parts:
- Oracle gap. For each test sentence, record which expert gives the lowest $F_0$ error. Compare oracle-selected error against the best fixed expert. This bounds what any weighting can achieve.
- What predicts the right horizon. Test whether the learned weights correlate with interpretable properties: questions, dialogue turns, paragraph boundaries, speaker changes in dialogue.
- Multiplicity. Train each horizon with three seeds and measure seed-to-seed disagreement in per-sentence $F_0$ contours. This quantifies the arbitrariness a single checkpoint hides and lets us separate the variance-reduction benefit from the horizon-selection benefit.
A known risk is that averaging pitch contours flattens expressiveness. We therefore track per-sentence $F_0$ standard deviation alongside error. We also run a variant that uses near-hard weights (low softmax temperature), in which blending approaches selection.
Baselines
The naive reference is a single context-free model: the zero-horizon expert alone, i.e. plain FastSpeech 2 with the same shared components. It establishes how much any context helps on our reimplementation.
The strongest existing method we can run on public data is the ATCE design from Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts. On J-MAC its succeeding-context variant reports $F_0$-RMSE 27.93 and Gross Pitch Error 12.57. We reimplement it on the same J-MAC split, acoustic backbone and vocoder as our system. We first confirm we land near those numbers before comparing. It gets its own hyperparameter search, of the same trial count as our weighting network's. Beating it means lower $F_0$-RMSE and GPE on our shared split, with the gap exceeding seed-to-seed standard deviation, plus a statistically significant win in a listening preference test.
On the 30-hour Mandarin audiobook corpus, Context-aware Coherent Speaking Style Prediction reports $F_0$ RMSE 62.17, style MSE 0.2524 and multi-sentence MOS 3.917. MSStyleTTS reports naturalness MOS 4.058 and paragraph-coherence MOS 3.867. That corpus appears to be in-house, so these figures serve as reference points only if access is obtained.
The in-network gate baseline is the key competitor for the ensembling claim. It is a single model containing all five context encoders, fused by a learned gate trained end-to-end, as in M2CI-Dubber's adaptor. Its parameter count and training compute match the ensemble's.
The uniform and seed-ensemble baselines isolate the key ingredient. The first averages the five horizon experts with equal weights. The second averages three seeds of the best fixed horizon, at equal inference cost. If input-conditioned horizon weighting matters, the proposal must beat both. If it beats only the uniform average, the gain is just trust calibration. If it matches the seed ensemble, the gain is just variance reduction.
Ablations remove, in turn, the cross-sentence weight smoothing, the causal-acoustic expert, and the held-out stacking. The stacking ablation fits weights on training-set predictions instead of held-out ones.
Why not just…?
Why not just use the single best fixed window? Because an averaged optimum can conceal per-sentence variation, and the oracle-gap analysis measures exactly this. The NN5 study found winner-take-all validation selection loses to combination on multi-step sequences (A review and comparison of strategies for multi-step ahead time series forecasting). The prosody-ensemble paper found weighted ensembles beat the best single recurrent model by 24% on $F_0$ (Weighted neural network ensemble models for speech prosody control). Neither result concerns context horizon, though. If our oracle gap is small, this alternative wins and we will say so.
Why not just let one network gate its own context branches, as M2CI-Dubber does? This is the real threat. An end-to-end gate is cheaper at inference and trained jointly. Our argument against it is specific to the degradation finding. A jointly trained model fits its gate on training data, where long-context branches look useful. Their failure appears only on held-out text, which is the failure mode the 64-versus-128 and 5-versus-9 results suggest. Stacking fits weights where that failure is visible. The compute-matched gate baseline settles the question directly. If it ties the ensemble, the contribution becomes the stacked-gate training procedure rather than separate experts, and the method is simplified accordingly.
Why not just average several seeds? The fairness results predict that seed averaging alone reduces arbitrary per-sentence prosody (Individual Arbitrariness and Group Fairness). Some of our gain could therefore be generic. The equal-cost seed ensemble, together with the measured seed-to-seed disagreement, separates the two sources of benefit.
Why not just merge the experts' weights into one checkpoint? Merging, as studied in Meta-Merging by Checkpoint Nowcasting, yields fixed parameters. It cannot change its behavior per sentence, which is the property under test. We include a parameter-averaged variant of the experts' variance predictors only as a cheap sanity check.
Decisive Checks
The main experiments run on J-MAC. We measure $F_0$-RMSE, GPE and $F_0$ frame error; per-sentence $F_0$ dynamic range; pitch discontinuity at sentence boundaries; naturalness MOS on single sentences; and a paragraph-level coherence MOS on consecutive passages. We also report inference cost relative to the single model, with and without the linear-recurrent context encoders.
The project hinges on three results, in order:
- Oracle gap. The per-sentence oracle over horizons must reduce $F_0$-RMSE substantially relative to the best fixed horizon, beyond seed noise. If it does not, the premise is false.
- Beating the matched baselines. The stacked, input-conditioned ensemble must recover a meaningful fraction of that oracle gap and beat both the uniform average and the equal-cost seed ensemble. Otherwise horizon adaptivity is not what helps.
- Coherence. The ensemble must beat the compute-matched in-network gate in paragraph-level coherence MOS without losing $F_0$ dynamic range. Otherwise the ensemble form is unnecessary and the simpler gated model should be preferred.
References
- Improving Speech Prosody of Audiobook Text-To-Speech Synthesis with Acoustic and Textual Contexts
- MSStyleTTS: Multi-Scale Style Modeling With Hierarchical Context Information for Expressive Speech Synthesis
- Weighted neural network ensemble models for speech prosody control
- Context-Aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis
- Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction
- A review and comparison of strategies for multi-step ahead time series\n forecasting based on the NN5 forecasting competition
- Individual Arbitrariness and Group Fairness
- Bidirectional linear recurrent models for sequence-level multisource fusion
- Meta-Merging by Checkpoint Nowcasting