Context-Aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis

Shun Lei, Yixuan Zhou, Liyang Chen, Zhiyong Wu, Shiyin Kang, Helen Meng

TL;DR

Researchers have developed a hierarchical transformer-based model that generates more natural and coherent speech for long-form audiobooks. By considering both the surrounding text and the actual speaking style of previous sentences, the model ensures that the "vibe" of the narration flows smoothly from one sentence to the next, avoiding the jarring transitions common in standard text-to-speech systems.

What Problem Does This Paper Address?

While modern Text-to-Speech (TTS) systems are excellent at producing clear, individual sentences, they often struggle with long-form content like audiobooks. In a book, the way a narrator speaks is influenced by the emotional arc and the context of the story.

Existing models often predict the speaking style (prosody, rhythm, and emotion) for each sentence in isolation or based solely on local text context. This leads to "prosodic inconsistency"—where the pitch or energy might jump abruptly between sentences, making the narration sound disjointed and robotic over time. The challenge is to achieve prosodic coherence: ensuring the synthesized speech is not just expressive, but also contextually appropriate and smooth across multiple sentences.

Methods

The authors propose a Context-aware Coherent Speaking Style Predictor integrated into a FastSpeech 2-based acoustic model. The architecture consists of three main components:

  1. Style Extractor: Based on the Global Style Token (GST) framework, this module extracts a "style embedding" from the actual mel-spectrogram of a recording. During training, this provides the "ground truth" style the model aims to predict.
  2. Hierarchical Context-aware Style Predictor: This is the core innovation. It uses a two-stage approach:
  3. Sentence Encoder: Uses XLNet to turn words into sentence-level context tokens.
  4. Fusion Context Encoder: A transformer that takes a sequence of context tokens from the current, past, and future sentences ($2N+1$ window). Crucially, it also takes the style embeddings of previous sentences as input.
  5. Mixture Attention Mask: To mimic human perception, the authors designed a specific attention mask. Text tokens can look at all other text tokens (bidirectional), but style tokens can only "look back" at previous style tokens and the surrounding text. This prevents the model from "cheating" by looking at future styles that wouldn't be available during real-time inference.

The model is trained using a knowledge distillation strategy: first training the style extractor, then using its outputs to teach the style predictor how to map text and history to the correct style embedding.

Datasets

The study used an internal Mandarin audiobook corpus.

  • Size: Approximately 30 hours of high-quality recordings.
  • Content: 87 chapters of a fiction book read by a professional male narrator.
  • Split: 14,500 audio clips, with 90% used for training and 10% for validation/testing.
  • Features: 80-dimensional mel-spectrograms sampled at 24 kHz.

Key Insights and Ideas

  • Speech-Side History Matters: Most models only look at the text of previous sentences. This paper argues that the actual sound (style embedding) of what was just spoken is a vital cue for what should come next.
  • Hierarchical Processing: By separating word-level encoding from sentence-level fusion, the model can capture both fine-grained semantic meaning and broad "super-sentential" prosody patterns (like pitch reset at the start of a new paragraph).
  • Coherence vs. Expressiveness: The authors found that while some models are very expressive (lots of variation), they lack coherence (the variation is random). Their approach balances both by anchoring the current sentence's style to the preceding ones.

Key Results

The proposed model was evaluated against standard FastSpeech 2 and a Hierarchical Context Encoder (HCE) baseline:

  • Subjective Metrics: In Mean Opinion Score (MOS) tests, the proposed model scored 3.990 for single sentences and 3.917 for multi-sentence sequences, significantly outperforming the HCE baseline (3.761) in multi-sentence coherence.
  • ABX Preference: In head-to-head comparisons, listeners preferred the proposed model over HCE by a margin of 24.3% for multi-sentence tests.
  • Objective Metrics: The model achieved the lowest Style MSE (0.2524) and F0 RMSE (62.17), indicating that its predicted pitch contours and style embeddings were much closer to the original human recordings than the baselines.
  • Ablation Studies: Removing the "previous style information" caused the biggest drop in performance, proving that hearing the "history" of the speech is essential for coherence.

Limitations and Future Work

  • Inference Dependency: Because the model relies on the style of previous sentences, errors in one sentence could potentially propagate to the next (though the paper suggests the text context helps mitigate this).
  • Language Specificity: The testing was limited to a single-speaker Mandarin dataset. Future work could explore how this generalizes to multi-speaker datasets or other languages with different prosodic structures (like English).
  • Long-term Memory: While the model uses a window of $2N+1$ sentences, very long-range dependencies (e.g., a character's specific "voice" returning after several pages) might require even more sophisticated memory mechanisms.

Summary updated

Abstract

Recent advances in text-to-speech have significantly improved the expressiveness of synthesized speech. However, it is still challenging to generate speech with contextually appropriate and coherent speaking style for multi-sentence text in audiobooks. In this paper, we propose a context-aware coherent speaking style prediction method for audiobook speech synthesis. To predict the style embedding of the current utterance, a hierarchical transformer-based context-aware style predictor with a mixture attention mask is designed, considering both text-side context information and speech-side style information of previous speeches. Based on this, we can generate long-form speech with coherent style and prosody sentence by sentence. Objective and subjective evaluations on a Mandarin audiobook dataset demonstrate that our proposed model can generate speech with more expressive and coherent speaking style than baselines, for both single-sentence and multi-sentence test.
BibTeX
@inproceedings{lei2023context,
  author = {Shun Lei and Yixuan Zhou and Liyang Chen and Zhiyong Wu and Shiyin Kang and Helen Meng},
  title = {{Context-Aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095866},
}

Loading related content...

Figure