Context-aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis

Shun Lei, Yixuan Zhou, Liyang Chen, Zhiyong Wu, Shiyin Kang, Helen Meng

Abstract

Recent advances in text-to-speech (TTS) have significantly improved the expressiveness of synthesized speech. However, generating contextually appropriate and coherent speaking styles for multi-sentence text in audiobooks remains a challenge. This paper proposes a context-aware coherent speaking style prediction method for audiobook speech synthesis. We introduce a hierarchical transformer-based context-aware style predictor with a mixture attention mask, considering both text-side context and speech-side style information from previous utterances. Objective and subjective evaluations on a Mandarin audiobook dataset demonstrate that the proposed model generates speech with a more expressive and coherent speaking style than existing baselines for both single-sentence and multi-sentence content.

Introduction

While modern TTS systems produce high-quality individual sentences, they often struggle with the long-form content typical of audiobooks. In narrative text, a narrator's delivery is influenced by emotional arcs and broader context. Standard models often predict prosody, rhythm, and emotion for each sentence in isolation or based only on local text, leading to "prosodic inconsistency." This results in abrupt jumps in pitch or energy between sentences, making the narration sound disjointed. The goal of this research is to achieve prosodic coherence: ensuring synthesized speech is expressive, contextually appropriate, and smooth across multiple sentences.

Methodology

The proposed system integrates a Context-aware Coherent Speaking Style Predictor into a FastSpeech 2-based acoustic model. The architecture, illustrated in Figure 1, comprises three primary components:

  1. Style Extractor: Utilizing the Global Style Token (GST) framework, this module extracts a style embedding from the ground-truth mel-spectrogram during training to serve as the target for prediction.
  2. Hierarchical Context-aware Style Predictor: This core innovation employs a two-stage approach. First, a Sentence Encoder (using XLNet) converts words into sentence-level context tokens. Second, a Fusion Context Encoder (a transformer) processes a sequence of context tokens from a window of current, past, and future sentences ($2N+1$). Crucially, it also incorporates the style embeddings of previous sentences as input.
  3. Mixture Attention Mask: To prevent the model from "cheating" by looking at future styles during training, a specific mask was designed. Text tokens maintain bidirectional attention, but style tokens can only attend to previous style tokens and the surrounding text context.
The overall architecture of the proposed context-aware TTS system, featuring the style predictor and extractor.

The model is trained using a knowledge distillation strategy, where the style extractor first learns to represent styles, and the style predictor is subsequently taught to map text and history to these embeddings.

Key Insights

  • Speech-Side History: Unlike models that only consider previous text, this approach argues that the actual sound (style embedding) of preceding sentences is a vital cue for maintaining coherence.
  • Hierarchical Processing: Separating word-level encoding from sentence-level fusion allows the model to capture both fine-grained semantics and broad "super-sentential" prosody patterns, such as the pitch reset often found at paragraph breaks.
  • Coherence vs. Expressiveness: The model balances high variation (expressiveness) with stability (coherence) by anchoring the current sentence's style to the preceding audio history.

Experimental Results

The model was evaluated using a 30-hour Mandarin audiobook corpus consisting of 87 chapters read by a professional male narrator.

Subjective Evaluation

In Mean Opinion Score (MOS) tests, the proposed model achieved 3.990 for single sentences and 3.917 for multi-sentence sequences, outperforming the Hierarchical Context Encoder (HCE) baseline (3.761) in multi-sentence coherence. As shown in Figure 2, head-to-head ABX preference tests further confirmed that listeners significantly preferred the proposed model over baselines in multi-sentence contexts.

Subjective ABX preference test results showing the proposed model's performance against FastSpeech2, HCE, and NP baselines.

Objective Evaluation

The model achieved the lowest Style MSE (0.2524) and F0 RMSE (62.17). These metrics indicate that the predicted pitch contours and style embeddings were significantly closer to the original human recordings than those produced by the baselines. Ablation studies revealed that removing previous style information caused the most significant performance drop, confirming that auditory history is essential for prosodic coherence.

Conclusion

This research demonstrates that incorporating both textual context and previous acoustic style embeddings significantly improves the coherence of long-form speech synthesis. While the current model relies on a fixed window of sentences, future work may explore more sophisticated memory mechanisms to handle very long-range dependencies, such as recurring character voices across an entire book. Additionally, testing on multi-speaker and multi-lingual datasets will be necessary to further validate the generalizability of this hierarchical approach.

Summary updated

Abstract

Recent advances in text-to-speech have significantly improved the expressiveness of synthesized speech. However, it is still challenging to generate speech with contextually appropriate and coherent speaking style for multi-sentence text in audiobooks. In this paper, we propose a context-aware coherent speaking style prediction method for audiobook speech synthesis. To predict the style embedding of the current utterance, a hierarchical transformer-based context-aware style predictor with a mixture attention mask is designed, considering both text-side context information and speech-side style information of previous speeches. Based on this, we can generate long-form speech with coherent style and prosody sentence by sentence. Objective and subjective evaluations on a Mandarin audiobook dataset demonstrate that our proposed model can generate speech with more expressive and coherent speaking style than baselines, for both single-sentence and multi-sentence test.
BibTeX
@inproceedings{lei2023context,
  author = {Shun Lei and Yixuan Zhou and Liyang Chen and Zhiyong Wu and Shiyin Kang and Helen Meng},
  title = {{Context-aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095866},
}
Figure