Abstract
PnG BERT is a novel encoder for neural Text-to-Speech (TTS) that integrates phoneme and grapheme representations into a unified, pre-trainable model. By employing a word-level masking strategy and alignment embeddings, the model captures deep linguistic context. Experimental results demonstrate that PnG BERT produces speech with naturalness and prosody comparable to professional human recordings, effectively bridging the gap between phonetic accuracy and semantic nuance.
Introduction
Modern neural TTS systems typically rely on either characters (graphemes) or phonetic transcriptions (phonemes) as input. However, both approaches have inherent limitations. Grapheme-only models often struggle with irregular English spellings and complex pronunciations. Conversely, phoneme-only models lose critical semantic context; for instance, homophones like "two" and "too" share phonemes but require distinct prosodic emphasis based on sentence structure. PnG BERT addresses these challenges by using a self-supervised BERT-style architecture to learn the intricate relationships between spelling and sound from massive unlabelled text corpora.
Methodology
The PnG BERT architecture augments the standard BERT framework in three primary ways to facilitate better alignment and representation learning:
- Dual-Segment Input: The model processes two parallel segments: a sequence of phonemes (IPA) and a sequence of graphemes (subword units).
- Word-Position Embeddings: To explicitly link phonemes to their corresponding graphemes, a fourth embedding layer is introduced. This "word-level alignment" assigns the same index to all tokens belonging to the same word, regardless of whether they are phonemes or graphemes.
- Consistent Word-Level Masking: During pre-training via Masked Language Modeling (MLM), if a word is selected for masking, both its phonemic and graphemic representations are masked. This prevents the model from trivially predicting a phoneme by looking at its spelling, forcing it to learn deeper linguistic context.
For the final TTS task, the PnG BERT encoder is integrated into a Non-Attentive Tacotron (NAT) architecture. While the model processes both inputs, only the hidden states from the phoneme positions are passed to the decoder, with graphemic information integrated via the self-attention mechanism.
Experimental Results
Naturalness and Prosody
In Side-by-Side (SxS) evaluations, PnG BERT achieved a score of 0.02 ± 0.10 against professional ground-truth recordings. Statistically, this indicates that human raters found no significant difference between the synthesized output and real human speech. The model achieved a Mean Opinion Score (MOS) of 4.47, outperforming the baseline NAT model (4.41) and versions without pre-training (4.30).
The "Trident" Attention Pattern
Analysis of the model's internal mechanics revealed a unique "trident" attention pattern across several self-attention heads. This visualization confirms that the model naturally learns to attend to a phoneme's own position, its immediate neighbors, and its corresponding graphemes simultaneously, validating the effectiveness of the word-position embeddings.
Discussion
The success of PnG BERT suggests that harder pre-training tasks—specifically word-level masking—lead to superior downstream performance. While this strategy results in lower accuracy during the pre-training phase compared to random masking, it forces the model to develop more robust semantic and syntactic representations. Raters specifically highlighted improvements in stress, tone, and inflection, particularly in complex "Hard lines" and questions, suggesting the model successfully leverages its language understanding to improve prosody.
Conclusion
PnG BERT represents a significant step forward in neural TTS by successfully combining the strengths of phonemic and graphemic inputs. Future work may explore extending this architecture to multilingual or code-switching scenarios and addressing the model's current limitations in generalizing to exceptionally long sequences.
Summary updated