Abstract
Recent advances in text-to-speech (TTS) have significantly improved the expressiveness of synthesized speech. However, generating contextually appropriate and coherent speaking styles for multi-sentence text in audiobooks remains a challenge. This paper proposes a context-aware coherent speaking style prediction method for audiobook speech synthesis. We introduce a hierarchical transformer-based context-aware style predictor with a mixture attention mask, considering both text-side context and speech-side style information from previous utterances. Objective and subjective evaluations on a Mandarin audiobook dataset demonstrate that the proposed model generates speech with a more expressive and coherent speaking style than existing baselines for both single-sentence and multi-sentence content.
Introduction
While modern TTS systems produce high-quality individual sentences, they often struggle with the long-form content typical of audiobooks. In narrative text, a narrator's delivery is influenced by emotional arcs and broader context. Standard models often predict prosody, rhythm, and emotion for each sentence in isolation or based only on local text, leading to "prosodic inconsistency." This results in abrupt jumps in pitch or energy between sentences, making the narration sound disjointed. The goal of this research is to achieve prosodic coherence: ensuring synthesized speech is expressive, contextually appropriate, and smooth across multiple sentences.
Methodology
The proposed system integrates a Context-aware Coherent Speaking Style Predictor into a FastSpeech 2-based acoustic model. The architecture, illustrated in Figure 1, comprises three primary components:
- Style Extractor: Utilizing the Global Style Token (GST) framework, this module extracts a style embedding from the ground-truth mel-spectrogram during training to serve as the target for prediction.
- Hierarchical Context-aware Style Predictor: This core innovation employs a two-stage approach. First, a Sentence Encoder (using XLNet) converts words into sentence-level context tokens. Second, a Fusion Context Encoder (a transformer) processes a sequence of context tokens from a window of current, past, and future sentences ($2N+1$). Crucially, it also incorporates the style embeddings of previous sentences as input.
- Mixture Attention Mask: To prevent the model from "cheating" by looking at future styles during training, a specific mask was designed. Text tokens maintain bidirectional attention, but style tokens can only attend to previous style tokens and the surrounding text context.
The model is trained using a knowledge distillation strategy, where the style extractor first learns to represent styles, and the style predictor is subsequently taught to map text and history to these embeddings.
Key Insights
- Speech-Side History: Unlike models that only consider previous text, this approach argues that the actual sound (style embedding) of preceding sentences is a vital cue for maintaining coherence.
- Hierarchical Processing: Separating word-level encoding from sentence-level fusion allows the model to capture both fine-grained semantics and broad "super-sentential" prosody patterns, such as the pitch reset often found at paragraph breaks.
- Coherence vs. Expressiveness: The model balances high variation (expressiveness) with stability (coherence) by anchoring the current sentence's style to the preceding audio history.
Experimental Results
The model was evaluated using a 30-hour Mandarin audiobook corpus consisting of 87 chapters read by a professional male narrator.
Subjective Evaluation
In Mean Opinion Score (MOS) tests, the proposed model achieved 3.990 for single sentences and 3.917 for multi-sentence sequences, outperforming the Hierarchical Context Encoder (HCE) baseline (3.761) in multi-sentence coherence. As shown in Figure 2, head-to-head ABX preference tests further confirmed that listeners significantly preferred the proposed model over baselines in multi-sentence contexts.
Objective Evaluation
The model achieved the lowest Style MSE (0.2524) and F0 RMSE (62.17). These metrics indicate that the predicted pitch contours and style embeddings were significantly closer to the original human recordings than those produced by the baselines. Ablation studies revealed that removing previous style information caused the most significant performance drop, confirming that auditory history is essential for prosodic coherence.
Conclusion
This research demonstrates that incorporating both textual context and previous acoustic style embeddings significantly improves the coherence of long-form speech synthesis. While the current model relies on a fixed window of sentences, future work may explore more sophisticated memory mechanisms to handle very long-range dependencies, such as recurring character voices across an entire book. Additionally, testing on multi-speaker and multi-lingual datasets will be necessary to further validate the generalizability of this hierarchical approach.
Summary updated