Who Needs Words? Lexicon-Free Speech Recognition

Tatiana Likhomanenko, Gabriel Synnaeve, Ronan Collobert

Abstract

Traditional Automatic Speech Recognition (ASR) systems rely on a fixed lexicon and word-level language models, creating a bottleneck for out-of-vocabulary (OOV) words. This paper demonstrates that character-based convolutional language models (ConvLMs) with large receptive fields can match the performance of word-based systems. We show that lexicon-free decoding not only achieves competitive Word Error Rates (WER) on standard benchmarks but also significantly outperforms lexicon-based systems on utterances containing OOV words.

The Lexicon Bottleneck

Most ASR systems are constrained by a dictionary. They use a lexicon—a predefined list of valid words—and a word-level language model to guide transcription. This architecture fails when encountering OOV words like new technical terms or proper nouns. In such cases, the system typically "hallucinates" the phonetically closest known word. While character-based systems avoid this constraint, they have historically struggled to capture the long-distance semantic context that whole words provide.

Methodology: Deep Character Context

The authors propose an end-to-end pipeline operating entirely at the character level, consisting of three core components:

  1. Acoustic Model (AM): A 1D Gated Convolutional Neural Network (GCNN) trained with the Auto Segmentation Criterion (ASG). It maps audio features to 31 graphemes (alphabet, punctuation, and a silence token).
  2. Language Model (LM): The "secret sauce" is the use of deep Convolutional LMs. By increasing the network depth, the authors achieved a receptive field of up to 81 characters. This allows the model to capture context comparable to traditional word-based models.
  3. Lexicon-Free Decoding: A modified beam-search decoder scores transcriptions using acoustic scores, character-LM scores, and penalties for word/silence length. Without a lexicon restriction, the system can generate any sequence of characters.

Performance and Accuracy

The system was evaluated on the Wall Street Journal (WSJ) and LibriSpeech datasets. On LibriSpeech, the lexicon-free GCNN-20B model demonstrated that it could match or even slightly exceed the performance of lexicon-constrained models.

Word and Character Error Rates on LibriSpeech across different models and lexicon settings.

On the WSJ dataset, the lexicon-free model achieved a 3.6% WER, matching the performance of the same model when constrained by a lexicon. Crucially, for utterances containing OOV words, the lexicon-free approach showed a massive relative improvement, achieving 18.3% WER compared to 22.2% for a standard word-based 4-gram model.

Efficiency and Decoding

A surprising finding was the efficiency of character-level decoding. Despite the lack of word-level constraints, character-based models require a smaller effective beam size to maintain accuracy.

Distribution of beam sizes required for different language models; character-based models stay on track with smaller beams.

While word-level decoders often need to track 2,000–8,000 hypotheses, the character-based model remains effective with a beam size around 2,000. This suggests that the character LM provides a strong enough signal to prune the search space effectively.

Conclusion

This research proves that lexicon-free, character-level modeling is a robust alternative to traditional word-based ASR. By leveraging large character contexts through deep convolutional architectures, these systems overcome the OOV problem without sacrificing accuracy on standard vocabulary. Future work may involve scaling to more complex writing systems and exploring Transformer-based architectures to further extend context.

Summary updated

Abstract

Lexicon-free speech recognition naturally deals with the problem of out-of-vocabulary (OOV) words. In this paper, we show that character-based language models (LM) can perform as well as word-based LMs for speech recognition, in word error rates (WER), even without restricting the decoding to a lexicon. We study character-based LMs and show that convolutional LMs can effectively leverage large (character) contexts, which is key for good speech recognition performance downstream. We specifically show that the lexicon-free decoding performance (WER) on utterances with OOV words using character-based LMs is better than lexicon-based decoding, both with character or word-based LMs.
BibTeX
@inproceedings{likhomanenko2019who,
  author = {Tatiana Likhomanenko and Gabriel Synnaeve and Ronan Collobert},
  title = {{Who Needs Words? Lexicon-Free Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=BWx2VzStY1K},
}
Figure