On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition

Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen

TL;DR

This paper challenges the traditional wisdom that phoneme-based modeling is superior for speech recognition when training data is scarce. The authors demonstrate that "end-to-end" units like graphemes and word-pieces consistently outperform phonemes in sequence-to-sequence models across all data scales, though combining these different units through rescoring can yield significant performance gains.

What Problem Does This Paper Address?

In traditional Automatic Speech Recognition (ASR), systems are "hybrid": they use a hand-crafted lexicon to map acoustic signals to phonemes, which are then mapped to words. For decades, the rule of thumb was that phonemes are essential for languages like English where spelling doesn't always match pronunciation, especially when you don't have much training data.

However, modern "Sequence-to-Sequence" (seq2seq) models—like the Listen, Attend, and Spell (LAS) architecture—can map audio directly to characters (graphemes) or sub-word units (word-pieces) without a lexicon. This paper investigates whether the old rule still applies: Do seq2seq models actually need phonemes to perform well, or have end-to-end units rendered them obsolete even in data-constrained scenarios?

Methods

The researchers compared three types of modeling units within the LAS (Listen, Attend, and Spell) framework:

  1. Phonemes: Requiring a pronunciation lexicon and an external n-gram language model for decoding.
  2. Graphemes: Individual characters (a, b, c...).
  3. Word-pieces: Sub-word units (e.g., "playing" might be "play" + "##ing") determined by data frequency.

Beyond simple comparison, they explored Model Complementarity. They used N-best list rescoring, where a hypothesis generated by a word-piece model is "double-checked" and re-ranked by a phoneme or grapheme model. They also experimented with a novel Auxiliary Decoder architecture—a single model that predicts graphemes as its primary task but has a secondary "sidecar" decoder predicting phonemes to help guide the internal representations.

Datasets

The study used the LibriSpeech corpus, a standard benchmark of read English speech. To test the impact of data volume, they evaluated models on three subsets:

  • 100 hours: Small/Limited data.
  • 460 hours: Medium data.
  • 960 hours: Large/Full dataset.

Key Insights and Ideas

  • The Death of the "Phoneme Advantage": Even at the 100-hour mark, grapheme and word-piece models outperformed phoneme-based models. This suggests that the attention mechanism in seq2seq models is powerful enough to learn the complex mapping between English audio and orthography without needing an explicit phonetic "middleman."
  • Diversity is a Strength: Word-piece models produce a much more diverse set of "guesses" (N-best hypotheses) than phoneme models. The "Oracle WER" (the error rate if you always picked the best guess in the list) was significantly lower for word-pieces.
  • The Homophone Trap: Phoneme models struggle with homophones (words that sound the same but are spelled differently, like "shoes" vs. "shews"). Because the phoneme model sees them as identical, it relies entirely on the external language model to guess the right one, which is often less effective than the internal "spelling" logic of a word-piece model.
  • Better Together: While word-piece models are the strongest standalone performers, they make different types of mistakes than phoneme models. Rescoring word-piece outputs with a phoneme model provided a "best of both worlds" boost.

Key Results

  • Standalone Performance: On the full 960hr "test-clean" set, the Word-piece model achieved a 4.7% Word Error Rate (WER), significantly beating the Phoneme model's 6.2%.
  • Data Scaling: Even on the limited 100hr set, Word-pieces (12.9% WER) beat Phonemes (14.3% WER).
  • Rescoring Gains: By rescoring the Word-piece N-best list with both phoneme and grapheme models, the authors reduced the WER from 4.7% to 4.3%, a relative improvement of about 9%.
  • Language Model Fusion: Combining the word-piece model with an external LSTM language model further pushed the performance to 3.6% WER on test-clean.

Limitations and Future Work

The authors noted that while the phoneme model helps in rescoring, it is inherently limited by its reliance on a fixed lexicon. If a word isn't in the dictionary, the phoneme model can't help.

Additionally, the Auxiliary Decoder (the multi-task model) didn't perform as well as independent rescoring. The authors suggest that future work should look into more sophisticated ways to integrate these units, perhaps within streaming architectures like the RNN-Transducer (RNN-T), which are more suitable for real-time applications than the LAS models used here.

Summary updated

Abstract

In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.
BibTeX
@inproceedings{irie2019choice,
  author = {Kazuki Irie and Rohit Prabhavalkar and Anjuli Kannan and Antoine Bruguier and David Rybach and Patrick Nguyen},
  title = {{On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=GpX2BEfcATy},
}

Loading related content...

Figure