Abstract
Lexicon-free speech recognition naturally deals with the problem of out-of-vocabulary (OOV) words. In this paper, we show that character-based language models (LM) can perform as well as word-based LMs for speech recognition, in word error rates (WER), even without restricting the decoding to a lexicon. We study character-based LMs and show that convolutional LMs can effectively leverage large (character) contexts, which is key for good speech recognition performance downstream. We specifically show that the lexicon-free decoding performance (WER) on utterances with OOV words using character-based LMs is better than lexicon-based decoding, both with character or word-based LMs.
Introduction
Historically, the most competitive Automatic Speech Recognition (ASR) systems have relied on a rigid pipeline: an acoustic model (AM) that predicts phonetic or character probabilities, paired with a word-level language model (LM) and a fixed lexicon (dictionary). During decoding, the system is constrained to output only words present in this lexicon.
This creates a significant bottleneck: Out-of-Vocabulary (OOV) words. If a speaker uses a novel name, a newly coined tech term, or slang not in the dictionary, a traditional ASR system literally cannot transcribe it, instead force-fitting the closest sounding known word. While "lexicon-free" approaches (predicting character-by-character without a dictionary) exist, they have historically suffered from higher complexity and significantly worse Word Error Rates (WER) compared to their lexicon-constrained counterparts. This paper aims to close that performance gap.
Methodology
The authors build an end-to-end ASR system using entirely character-level components:
- Acoustic Model (AM): A 1D gated convolutional neural network trained to map audio features (log-mel filterbanks) directly to 31 graphemes (the English alphabet, punctuation, and a silence/word-boundary token). It is trained using the Auto Segmentation Criterion (ASG).
- Language Model (LM): Instead of standard word-level LMs, the system uses character-level LMs. Specifically, deep Gated Convolutional Language Models (ConvLMs) are used to capture enough context to rival word-level models. These architectures (up to 20 convolutional layers) result in massive receptive fields of up to 81 characters.
- Beam-Search Decoder: A custom beam-search decoder optimizes a combined score of the AM likelihood, the LM likelihood, and penalties for word length and silence. Crucially, this decoder operates lexicon-free, meaning it explores character sequences without checking them against a predefined list of valid words.
Experimental Results
The models were trained and evaluated on two standard English ASR benchmarks: Wall Street Journal (WSJ) and Librispeech (1000 hours).
Quantitative Performance
The character-level ConvLM achieved state-of-the-art results. On the WSJ nov92 test set, the lexicon-free character ConvLM achieved a 3.6% WER, outperforming the word-level ConvLM baseline (4.3% WER). On Librispeech, the lexicon-free model achieved a 3.5% WER on the test-clean set, nearly identical to the word-level baseline of 3.4%.
Handling Out-of-Vocabulary Words
The primary advantage of the lexicon-free approach is its ability to transcribe OOV words. On Librispeech utterances specifically containing OOV words, the lexicon-free model dropped the WER from 9.9% (word-level LM) to 7.7%. Qualitative analysis shows the model successfully transcribing complex, unseen names like "pekaiah" and "arkadyevitch," which word-level models failed to recognize.
Decoding Efficiency
Surprisingly, lexicon-free decoding is efficient. The authors found that character-level LMs require a smaller effective beam size (around 2000) compared to word-level models, which often need beam sizes of 2000–8000 to avoid dropping good hypotheses.
Conclusion
This research demonstrates that character-level convolutional LMs can match the performance of word-level models while providing the critical benefit of lexicon-free decoding. By leveraging large receptive fields, these models understand word boundaries and grammar effectively, allowing for the accurate transcription of out-of-vocabulary words without degrading performance on common vocabulary. Future work involves scaling these models to even larger datasets and exploring multi-lingual settings where OOV rates are naturally higher.
Summary updated