Transformer Based Grapheme-to-Phoneme Conversion

Sevinj Yolchuyeva, Géza Németh, Bálint Gyires-TóthInterspeech 2019

TL;DR

This paper introduces the Transformer architecture for Grapheme-to-Phoneme (G2P) conversion, a crucial step in speech technologies. The Transformer-based model achieves state-of-the-art results, outperforming previous recurrent and convolutional approaches in accuracy while also being significantly smaller and faster to train.

What Problem Does This Paper Address?

Grapheme-to-Phoneme (G2P) conversion is the process of transforming written text (graphemes) into their corresponding spoken sounds (phonemes). This is a fundamental component for technologies like Text-to-Speech (TTS) and Automatic Speech Recognition (ASR). While various methods have been explored, including rule-based systems, joint sequence models, and recurrent/convolutional neural networks, there's a continuous drive to improve accuracy, efficiency, and model size. This paper specifically investigates whether the Transformer architecture, which has revolutionized Natural Language Processing (NLP) tasks like machine translation, can offer similar benefits for G2P conversion.

Methods

The core of this paper is the application of the Transformer architecture to G2P conversion. The Transformer is an encoder-decoder model that relies entirely on self-attention mechanisms, eschewing recurrent or convolutional layers.

  • Encoder-Decoder Structure: The model follows the standard Transformer encoder-decoder paradigm. The encoder processes the input grapheme sequence, and the decoder generates the output phoneme sequence.
  • Self-Attention: The key innovation is the use of multi-head self-attention. This mechanism allows the model to weigh the importance of different parts of the input sequence when processing each element, enabling it to capture long-range dependencies without sequential processing.
  • Positional Encoding: Since the Transformer lacks inherent sequential processing, positional encodings are added to the input and output embeddings to provide the model with information about the order of graphemes and phonemes.
  • Multi-Head Attention: This technique involves running several attention mechanisms in parallel, each learning different representations. The results are then concatenated and projected, allowing the model to attend to information from various representation subspaces.
  • Model Variations: The authors experimented with different depths for the Transformer, specifically using 3, 4, and 5 encoder and decoder layers (referred to as Transformer 3x3, 4x4, and 5x5).

Datasets

The experiments were conducted on two standard datasets for G2P research:

  • CMUDict: This dataset contains 106,837 words for training, 12,000 for testing, and 2,670 for validation. It includes 27 graphemes and 41 phonemes.
  • NetTalk: This dataset consists of 14,851 words for training and 4,951 for testing. It uses 26 graphemes and 52 phonemes.

The paper notes that the train/test splits used are consistent with previous research, allowing for direct comparison of results.

Key Insights and Ideas

  • Transformer's Efficacy in G2P: The Transformer architecture, known for its success in NLP, proves highly effective for G2P conversion, achieving state-of-the-art performance.
  • Beyond Recurrence and Convolutions: The paper demonstrates that G2P conversion can be successfully tackled without relying on traditional recurrent or convolutional layers, which were dominant in previous G2P models.
  • Model Size Matters: Interestingly, increasing the number of Transformer layers (from 4 to 5) did not necessarily lead to better performance and resulted in a larger model. This suggests that for G2P, a moderate depth might be optimal, and the complexity of NMT is not always required.
  • Efficiency Gains: Transformer models are shown to be significantly smaller in terms of parameters compared to previous state-of-the-art convolutional and recurrent models, leading to faster training times.
  • PER vs. WER Trade-off: The paper highlights an interesting observation where models with lower Phoneme Error Rate (PER) might not always have the lowest Word Error Rate (WER), suggesting that the nature of errors (e.g., single phoneme errors vs. multiple) can impact WER differently.

Key Results

  • State-of-the-Art Performance: The Transformer 4x4 model achieved competitive Phoneme Error Rates (PER) and Word Error Rates (WER) on both CMUDict and NetTalk datasets.
  • Outperforming Previous Methods: The Transformer-based approach significantly surpassed previous recurrent methods in terms of both PER and WER. It also outperformed convolutional-based approaches in terms of WER.
  • Model Size Reduction: The Transformer 4x4 model, with 1.95M parameters, is considerably smaller than previous state-of-the-art models like the encoder CNN with residual connections and Bi-LSTM decoder (14.5M parameters).
  • Faster Training: The reduced model size and efficient architecture contribute to faster training and validation times. For instance, the Transformer 4x4 model's training time per epoch was 98 seconds on CMUDict.

Limitations and Future Work

  • Model Depth Optimization: The study found that increasing the number of encoder-decoder layers beyond 4 did not yield significant improvements and even led to worse performance in some cases. Further research could explore optimal layer configurations more deeply.
  • Beam Search Not Used: The authors did not employ beam search during inference, opting for greedy decoding. Incorporating beam search might further improve performance.
  • Future Application in TTS: The authors express their intention to explore the application of the proposed Transformer-based G2P method in end-to-end Text-to-Speech (TTS) synthesis systems.

Summary updated

BibTeX
@inproceedings{yolchuyeva2019transformer,
  author = {Sevinj Yolchuyeva and Géza Németh and Bálint Gyires-Tóth},
  title = {{Transformer Based Grapheme-to-Phoneme Conversion}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure