Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions

Yinghao Aaron Li, Cong Han, Xilin Jiang, Nima Mesgarani

Abstract

Large-scale pre-trained language models have been shown to be helpful in improving the naturalness of text-to-speech (TTS) models by enabling them to produce more naturalistic prosodic patterns. However, these models are usually word-level or sup-phoneme-level and jointly trained with phonemes, making them inefficient for the downstream TTS task where only phonemes are needed. In this work, we propose a phoneme-level BERT (PL-BERT) with a pretext task of predicting the corresponding graphemes along with the regular masked phoneme predictions. Subjective evaluations show that our phoneme-level BERT encoder has significantly improved the mean opinion scores (MOS) of rated naturalness of synthesized speech compared with the state-of-the-art (SOTA) StyleTTS baseline on out-of-distribution (OOD) texts.

Introduction

Modern Text-to-Speech (TTS) systems often struggle with prosody—the patterns of stress, intonation, and rhythm that make speech sound human. While these systems can achieve high-fidelity audio, they frequently suffer from a "non-native speaker" effect, where sounds are correct but linguistic nuances are missed. This is largely due to a lack of deep contextual understanding.

Existing solutions leverage large-scale pre-trained language models like BERT. However, these models typically operate at the word or character level, creating a mismatch with TTS models that operate on phonemes. Bridging this gap often involves complex "sub-phoneme" units or computationally expensive joint training, which can be slow and prone to failure on out-of-vocabulary words.

Methods

The researchers introduced PL-BERT, a lightweight model based on the ALBERT architecture that operates directly at the phoneme level. The core innovation is a dual-objective pre-training strategy:

  1. Masked Language Modeling (MLM): The model predicts masked phonemes using whole-word masking, forcing it to rely on broader linguistic context.
  2. Phoneme-to-Grapheme (P2G) Prediction: For every phoneme input, the model is tasked with predicting the corresponding word (grapheme). This enables the phoneme-level encoder to learn semantic meaning and word-level structure without requiring a massive, specialized vocabulary.
PL-BERT architecture showing phoneme and grapheme prediction targets.

Datasets

PL-BERT was pre-trained on the English Wikipedia corpus (74M sentences) using International Phonetic Alphabet (IPA) phonemes. For TTS fine-tuning, the LJSpeech dataset (24 hours of single-speaker audio) was used. To evaluate generalization, the model was tested on "out-of-distribution" (OOD) texts from the Gutenberg books dataset, which were never seen during training.

Results and Discussion

The integration of PL-BERT into the StyleTTS framework yielded significant improvements in speech naturalness, particularly for unseen text.

Comparison of MOS and CMOS scores across different models on in-distribution and OOD text.

Key findings include:

  • Superior OOD Performance: On the Gutenberg dataset, StyleTTS with PL-BERT achieved a Mean Opinion Score (MOS) of 3.64, significantly outperforming both the baseline StyleTTS (3.49) and the previous state-of-the-art MP-BERT (3.55).
  • Efficiency: By staying purely at the phoneme level and avoiding large BPE-based vocabularies, PL-BERT is 1.05x faster than MP-BERT and more robust to unseen words.
  • Ablation Success: Removing the P2G task led to a performance drop, confirming that predicting graphemes is essential for capturing the semantic cues necessary for natural prosody.

Conclusion

PL-BERT demonstrates that a phoneme-level approach, augmented with grapheme prediction, can effectively bridge the gap between linguistic understanding and acoustic synthesis. While a performance gap remains between in-distribution and OOD texts, this work suggests that future TTS research should prioritize OOD evaluations to better reflect real-world utility.

Summary updated

Abstract

Large-scale pre-trained language models have been shown to be helpful in improving the naturalness of text-to-speech (TTS) models by enabling them to produce more naturalistic prosodic patterns. However, these models are usually word-level or sup-phoneme-level and jointly trained with phonemes, making them inefficient for the downstream TTS task where only phonemes are needed. In this work, we propose a phoneme-level BERT (PL-BERT) with a pretext task of predicting the corresponding graphemes along with the regular masked phoneme predictions. Subjective evaluations show that our phoneme-level BERT encoder has significantly improved the mean opinion scores (MOS) of rated naturalness of synthesized speech compared with the state-of-the-art (SOTA) StyleTTS baseline on out-of-distribution (OOD) texts.
BibTeX
@inproceedings{li2023phoneme,
  author = {Yinghao Aaron Li and Cong Han and Xilin Jiang and Nima Mesgarani},
  title = {{Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097074},
}
Figure