Who Needs Words? Lexicon-Free Speech Recognition

Tatiana Likhomanenko, Gabriel Synnaeve, Ronan Collobert

Abstract

Lexicon-free speech recognition naturally deals with the problem of out-of-vocabulary (OOV) words. In this paper, we show that character-based language models (LM) can perform as well as word-based LMs for speech recognition, in word error rates (WER), even without restricting the decoding to a lexicon. We study character-based LMs and show that convolutional LMs can effectively leverage large (character) contexts, which is key for good speech recognition performance downstream. We specifically show that the lexicon-free decoding performance (WER) on utterances with OOV words using character-based LMs is better than lexicon-based decoding, both with character or word-based LMs.

Introduction

Historically, the most competitive Automatic Speech Recognition (ASR) systems have relied on a rigid pipeline: an acoustic model (AM) that predicts phonetic or character probabilities, paired with a word-level language model (LM) and a fixed lexicon (dictionary). During decoding, the system is constrained to output only words present in this lexicon.

This creates a significant bottleneck: Out-of-Vocabulary (OOV) words. If a speaker uses a novel name, a newly coined tech term, or slang not in the dictionary, a traditional ASR system literally cannot transcribe it, instead force-fitting the closest sounding known word. While "lexicon-free" approaches (predicting character-by-character without a dictionary) exist, they have historically suffered from higher complexity and significantly worse Word Error Rates (WER) compared to their lexicon-constrained counterparts. This paper aims to close that performance gap.

Methodology

The authors build an end-to-end ASR system using entirely character-level components:

  • Acoustic Model (AM): A 1D gated convolutional neural network trained to map audio features (log-mel filterbanks) directly to 31 graphemes (the English alphabet, punctuation, and a silence/word-boundary token). It is trained using the Auto Segmentation Criterion (ASG).
  • Language Model (LM): Instead of standard word-level LMs, the system uses character-level LMs. Specifically, deep Gated Convolutional Language Models (ConvLMs) are used to capture enough context to rival word-level models. These architectures (up to 20 convolutional layers) result in massive receptive fields of up to 81 characters.
  • Beam-Search Decoder: A custom beam-search decoder optimizes a combined score of the AM likelihood, the LM likelihood, and penalties for word length and silence. Crucially, this decoder operates lexicon-free, meaning it explores character sequences without checking them against a predefined list of valid words.

Experimental Results

The models were trained and evaluated on two standard English ASR benchmarks: Wall Street Journal (WSJ) and Librispeech (1000 hours).

Quantitative Performance

The character-level ConvLM achieved state-of-the-art results. On the WSJ nov92 test set, the lexicon-free character ConvLM achieved a 3.6% WER, outperforming the word-level ConvLM baseline (4.3% WER). On Librispeech, the lexicon-free model achieved a 3.5% WER on the test-clean set, nearly identical to the word-level baseline of 3.4%.

Table 4: Word Error Rate (WER) and Character Error Rate (CER) on Librispeech. Lexicon-free character models (no lexicon) achieve performance parity with word-based models.

Handling Out-of-Vocabulary Words

The primary advantage of the lexicon-free approach is its ability to transcribe OOV words. On Librispeech utterances specifically containing OOV words, the lexicon-free model dropped the WER from 9.9% (word-level LM) to 7.7%. Qualitative analysis shows the model successfully transcribing complex, unseen names like "pekaiah" and "arkadyevitch," which word-level models failed to recognize.

Figure 6: Qualitative transcription examples comparing Target (T), Word LM (W), Character LM with lexicon (C), and Lexicon-Free Character LM (F).

Decoding Efficiency

Surprisingly, lexicon-free decoding is efficient. The authors found that character-level LMs require a smaller effective beam size (around 2000) compared to word-level models, which often need beam sizes of 2000–8000 to avoid dropping good hypotheses.

Conclusion

This research demonstrates that character-level convolutional LMs can match the performance of word-level models while providing the critical benefit of lexicon-free decoding. By leveraging large receptive fields, these models understand word boundaries and grammar effectively, allowing for the accurate transcription of out-of-vocabulary words without degrading performance on common vocabulary. Future work involves scaling these models to even larger datasets and exploring multi-lingual settings where OOV rates are naturally higher.

Summary updated

Abstract

Lexicon-free speech recognition naturally deals with the problem of out-of-vocabulary (OOV) words. In this paper, we show that character-based language models (LM) can perform as well as word-based LMs for speech recognition, in word error rates (WER), even without restricting the decoding to a lexicon. We study character-based LMs and show that convolutional LMs can effectively leverage large (character) contexts, which is key for good speech recognition performance downstream. We specifically show that the lexicon-free decoding performance (WER) on utterances with OOV words using character-based LMs is better than lexicon-based decoding, both with character or word-based LMs.
BibTeX
@inproceedings{likhomanenko2019who,
  author = {Tatiana Likhomanenko and Gabriel Synnaeve and Ronan Collobert},
  title = {{Who Needs Words? Lexicon-Free Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=BWx2VzStY1K},
}
Figure