Model Unit Exploration for Sequence-to-Sequence Speech Recognition

Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen

Abstract

In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.

Introduction

Traditional Automatic Speech Recognition (ASR) systems have long relied on a "hybrid" architecture, utilizing a hand-crafted lexicon to bridge acoustic signals and phonemes, which are then mapped to words. A long-standing heuristic in the field suggests that phonemes are indispensable for languages with irregular orthography, such as English, particularly in low-resource settings where data is scarce.

Modern sequence-to-sequence (seq2seq) architectures, such as the Listen, Attend, and Spell (LAS) framework, challenge this paradigm by enabling direct mapping from audio to characters (graphemes) or sub-word units (word-pieces). This study investigates whether the traditional reliance on phonemes remains justified in the context of attention-based models across varying data scales.

Methods

The study evaluates three primary modeling units within the LAS framework:

  1. Phonemes: Decoded using a pronunciation lexicon and an external n-gram language model.
  2. Graphemes: Individual characters (e.g., a, b, c).
  3. Word-pieces: Data-driven sub-word units determined by frequency.

To explore model complementarity, the researchers employed N-best list rescoring, where hypotheses from a primary word-piece model are re-ranked by auxiliary phoneme or grapheme models. Additionally, a novel Auxiliary Decoder architecture was proposed, which incorporates a secondary decoder to predict phonemes alongside the primary grapheme task, aiming to enrich internal representations.

The Auxiliary Decoder architecture integrates a secondary phoneme-predicting decoder to guide the primary model's representations.

Experimental Setup

Models were trained and evaluated on the LibriSpeech corpus across three data subsets to assess scaling behavior: 100 hours (limited), 460 hours (medium), and 960 hours (full dataset).

Results and Discussion

Performance of End-to-End Units

Contrary to traditional wisdom, grapheme and word-piece models consistently outperformed phoneme-based models, even with only 100 hours of training data. This indicates that the attention mechanism effectively learns complex audio-to-orthography mappings without explicit phonetic guidance.

Word Error Rate (WER) comparison for Grapheme and Word-Piece units across different model sizes and LibriSpeech subsets.

As shown in the results, the word-piece model with 180M parameters achieved the best standalone performance, reaching a 4.7% WER on the "test-clean" set. In comparison, the phoneme-based model achieved a 6.2% WER on the same task. Even on the limited 100hr set, word-pieces (12.9% WER) outperformed phonemes (14.3% WER).

Diversity and Complementarity

Word-piece models demonstrated greater hypothesis diversity compared to phoneme models, resulting in significantly lower "Oracle WERs" (the error rate if the best hypothesis in the N-best list is always selected). Furthermore, phoneme models were found to struggle with homophones (e.g., "shoes" vs. "shews"), as they rely solely on external language models to resolve phonetic ambiguities.

The study found significant gains through model fusion. Rescoring word-piece N-best lists with phoneme and grapheme models yielded a 9% relative improvement, reducing the WER from 4.7% to 4.3%. Integration with an external LSTM language model further improved performance to 3.6% WER on test-clean.

Conclusion

This research demonstrates that end-to-end units like word-pieces have largely superseded phonemes as the optimal modeling unit for seq2seq speech recognition, even in data-constrained environments. While phoneme models provide value in rescoring, their utility is capped by their dependence on fixed lexicons. Future work may explore more integrated multi-unit architectures and their application in streaming ASR systems.

Summary updated

Abstract

In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.
BibTeX
@inproceedings{irie2019model,
  author = {Kazuki Irie and Rohit Prabhavalkar and Anjuli Kannan and Antoine Bruguier and David Rybach and Patrick Nguyen},
  title = {{Model Unit Exploration for Sequence-to-Sequence Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=GpX2BEfcATy},
}
Figure