PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS

Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, Yonghui Wu

Abstract

PnG BERT is a novel encoder for neural Text-to-Speech (TTS) that integrates phoneme and grapheme representations into a unified, pre-trainable model. By employing a word-level masking strategy and alignment embeddings, the model captures deep linguistic context. Experimental results demonstrate that PnG BERT produces speech with naturalness and prosody comparable to professional human recordings, effectively bridging the gap between phonetic accuracy and semantic nuance.

Introduction

Modern neural TTS systems typically rely on either characters (graphemes) or phonetic transcriptions (phonemes) as input. However, both approaches have inherent limitations. Grapheme-only models often struggle with irregular English spellings and complex pronunciations. Conversely, phoneme-only models lose critical semantic context; for instance, homophones like "two" and "too" share phonemes but require distinct prosodic emphasis based on sentence structure. PnG BERT addresses these challenges by using a self-supervised BERT-style architecture to learn the intricate relationships between spelling and sound from massive unlabelled text corpora.

Methodology

The PnG BERT architecture augments the standard BERT framework in three primary ways to facilitate better alignment and representation learning:

  1. Dual-Segment Input: The model processes two parallel segments: a sequence of phonemes (IPA) and a sequence of graphemes (subword units).
  2. Word-Position Embeddings: To explicitly link phonemes to their corresponding graphemes, a fourth embedding layer is introduced. This "word-level alignment" assigns the same index to all tokens belonging to the same word, regardless of whether they are phonemes or graphemes.
  3. Consistent Word-Level Masking: During pre-training via Masked Language Modeling (MLM), if a word is selected for masking, both its phonemic and graphemic representations are masked. This prevents the model from trivially predicting a phoneme by looking at its spelling, forcing it to learn deeper linguistic context.
PnG BERT architecture for pre-training (a) and its integration as a TTS encoder (b).

For the final TTS task, the PnG BERT encoder is integrated into a Non-Attentive Tacotron (NAT) architecture. While the model processes both inputs, only the hidden states from the phoneme positions are passed to the decoder, with graphemic information integrated via the self-attention mechanism.

Experimental Results

Naturalness and Prosody

In Side-by-Side (SxS) evaluations, PnG BERT achieved a score of 0.02 ± 0.10 against professional ground-truth recordings. Statistically, this indicates that human raters found no significant difference between the synthesized output and real human speech. The model achieved a Mean Opinion Score (MOS) of 4.47, outperforming the baseline NAT model (4.41) and versions without pre-training (4.30).

The "Trident" Attention Pattern

Analysis of the model's internal mechanics revealed a unique "trident" attention pattern across several self-attention heads. This visualization confirms that the model naturally learns to attend to a phoneme's own position, its immediate neighbors, and its corresponding graphemes simultaneously, validating the effectiveness of the word-position embeddings.

Visualization of self-attention heads across layers, highlighting the 'trident' alignment pattern between phonemes and graphemes.

Discussion

The success of PnG BERT suggests that harder pre-training tasks—specifically word-level masking—lead to superior downstream performance. While this strategy results in lower accuracy during the pre-training phase compared to random masking, it forces the model to develop more robust semantic and syntactic representations. Raters specifically highlighted improvements in stress, tone, and inflection, particularly in complex "Hard lines" and questions, suggesting the model successfully leverages its language understanding to improve prosody.

Conclusion

PnG BERT represents a significant step forward in neural TTS by successfully combining the strengths of phonemic and graphemic inputs. Future work may explore extending this architecture to multilingual or code-switching scenarios and addressing the model's current limitations in generalizing to exceptionally long sequences.

Summary updated

Abstract

This paper introduces PnG BERT, a new encoder model for neural TTS. This model is augmented from the original BERT model, by taking both phoneme and grapheme representations of text as input, as well as the word-level alignment between them. It can be pre-trained on a large text corpus in a self-supervised manner, and fine-tuned in a TTS task. Experimental results show that a neural TTS model using a pre-trained PnG BERT as its encoder yields more natural prosody and more accurate pronunciation than a baseline model using only phoneme input with no pre-training. Subjective side-by-side preference evaluations show that raters have no statistically significant preference between the speech synthesized using a PnG BERT and ground truth recordings from professional speakers.
BibTeX
@inproceedings{jia2021png,
  author = {Ye Jia and Heiga Zen and Jonathan Shen and Yu Zhang and Yonghui Wu},
  title = {{PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS}},
  booktitle = {Interspeech},
  year = {2021},
  doi = {10.21437/interspeech.2021-1757},
}
Figure