Acoustic-dependent Phonemic Transcription for Text-to-speech Synthesis

Kévin Vythelingum, Yannick Estève, Olivier RosecInterspeech 2018

TL;DR

This paper tackles the crucial problem of accurately transcribing speech into phonemes for text-to-speech (TTS) systems. By combining acoustic information with text-based predictions, they achieve a significant reduction in phonemic transcription errors, paving the way for more cost-effective creation of high-quality TTS data.

What Problem Does This Paper Address?

Building a high-quality text-to-speech (TTS) system relies heavily on accurate phonemic transcriptions of speech data. These transcriptions are essential for segmenting speech into acoustic units and training the underlying models. Traditionally, phonemic transcriptions are generated automatically using grapheme-to-phoneme (G2P) conversion systems. However, these systems often struggle with speaker variability and can produce errors that are difficult and costly to correct manually. The core problem this paper addresses is how to automatically detect and reduce these G2P conversion errors by leveraging both the speech signal and textual information, thereby improving the overall accuracy of phonemic transcriptions and reducing the manual effort required for data annotation.

Methods

The authors explore several approaches to obtain "signal-dependent" phonemic transcriptions, meaning transcriptions that are informed by the actual speech signal, not just the text.

  • Neural Sequence-to-Sequence G2P Conversion: They employ an encoder-decoder architecture (similar to machine translation models) to convert sequences of characters into phonemes. This model is trained on sentence-level data and incorporates an attention mechanism to handle long sequences.
  • Acoustic Forced Alignment: This technique aligns a given pronunciation lexicon with the speech signal at the phoneme level. To improve its accuracy, they experiment with enriching the pronunciation lexicon by adding "n-best" hypotheses from G2P conversion systems.
  • End-to-End Phoneme Recognition: This approach uses a DeepSpeech 2 architecture to directly predict phoneme sequences from the acoustic signal, bypassing explicit G2P conversion. They also investigate incorporating sentence-level text embeddings (derived from their neural G2P model) into this system to provide textual context.
  • Error Detection Framework: The core contribution lies in using these different phonemic transcription systems to detect errors made by a primary G2P conversion system (their neural sequence-to-sequence model). They compare the outputs of the different systems to identify potential mismatches, flagging them as erroneous.

Datasets

The models were trained on internal French TTS datasets comprising approximately 50 hours of speech data from 9 speakers, segmented into 90,135 utterances. Evaluation was performed on a smaller internal French TTS dataset of approximately 1 hour of speech data from 3 speakers, segmented into 951 utterances. The speech data was manually transcribed at both word and phoneme levels.

Key Insights and Ideas

  • Signal Matters: The paper strongly reinforces the idea that incorporating acoustic information is crucial for accurate phonemic transcription, especially when dealing with speaker variability.
  • Neural G2P Outperforms: Their neural sequence-to-sequence G2P model significantly outperforms a traditional joint-sequence model, highlighting the power of deep learning for this task.
  • Forced Alignment Benefits from G2P Hypotheses: Enriching the lexicon for forced alignment with even just the top G2P hypothesis improves its performance. However, adding too many hypotheses can degrade performance.
  • End-to-End Recognition is Promising but Needs Context: While end-to-end phoneme recognition can work directly from speech, its accuracy is lower than G2P or forced alignment. Crucially, providing sentence-level text embeddings to the end-to-end model significantly improves its performance.
  • Combined Approach for Error Detection: The most significant finding is that combining forced alignment and end-to-end phoneme recognition provides the best results for error detection, achieving high recall (detecting most errors) while keeping the manual checking rate low.

Key Results

  • Phonemic Transcription:
    • Neural sequence-to-sequence G2P achieved a Phone Error Rate (PER) of 2.8%.
  • Forced alignment with the best G2P hypothesis enrichment achieved a PER of 4.3%.
  • End-to-end phoneme recognition with text embeddings achieved a PER of 9.6%.
  • Error Detection:
    • The best error detection system, a combination of forced alignment and end-to-end phoneme recognition, could detect up to 90.5% of G2P conversion errors while flagging only 15.8% of phonemes as potentially erroneous. This represents a substantial reduction in the manual effort needed to correct errors.

Limitations and Future Work

  • Limitations: The paper acknowledges that the end-to-end phoneme recognition accuracy is lower than G2P conversion and forced alignment. The error detection performance, while good, still requires a human annotator to check a portion of the data.
  • Future Work: The authors suggest exploring other use cases for multimodal phonemic transcription and grapheme-to-phoneme error detection. Further research could focus on improving the accuracy of individual transcription systems and exploring more sophisticated ways to combine information from different modalities.

Summary updated

BibTeX
@inproceedings{vythelingum2018acoustic,
  author = {Kévin Vythelingum and Yannick Estève and Olivier Rosec},
  title = {{Acoustic-dependent Phonemic Transcription for Text-to-speech Synthesis}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure