Abstract
Contextual automatic speech recognition (ASR), which biases recognition toward specific contexts like user contacts or playlists, remains a significant challenge for end-to-end (E2E) models. These models often struggle with rare named entities, particularly foreign proper nouns that are out-of-vocabulary (OOV) relative to the training data. While standard grapheme or wordpiece models fail to spell these unseen words correctly, phonemes offer a more acoustically salient representation. This work proposes a hybrid E2E model incorporating both English wordpieces and phonemes. By mapping foreign pronunciations into a shared English phoneme space, we perform contextual biasing at the phoneme level. Experimental results demonstrate that this approach achieves a 16% relative improvement over grapheme-only biasing and an 8% improvement over wordpiece-only biasing on foreign place name recognition, with minimal impact on general English tasks.
Introduction
Modern E2E ASR models, such as RNN-Transducers, typically map acoustic signals directly to sub-word units like wordpieces. While efficient for high-frequency vocabulary, these models often fail on rare named entities—specifically foreign geographic locations (e.g., "Créteil")—because they lack the internal spelling logic for words virtually unseen during training.
Contextual biasing aims to address this by injecting a list of expected words into the decoding process. However, when the model has never encountered the orthography of these foreign words, traditional grapheme-based biasing is ineffective. This research explores using phonemes as an acoustic bridge to enable cross-lingual recognition within an English-centric model.
Methodology
The proposed system utilizes a three-pronged strategy to bridge the gap between English training data and foreign-word recognition:
- Hybrid Wordpiece-Phoneme Model: The model is trained to output either standard wordpieces or sequences of phonemes. During training, rare words are randomly decomposed into phoneme sequences, teaching the model to "sound out" unfamiliar acoustic patterns.
- Cross-Lingual Phoneme Mapping: Foreign names are converted into their native phonemes (e.g., French) and then mapped to the nearest sounding English phonemes. This allows an English-trained model to recognize the acoustic signature of a foreign word.
- Phoneme-Level Contextual Biasing: A Weighted Finite State Transducer (WFST) is used during decoding to bias the model toward specific phoneme sequences. These sequences are then mapped back to the correct foreign orthography for the final output.
Experimental Evaluation
The model was trained on 35 million English utterances (approximately 27,500 hours) of anonymized traffic data. Evaluation was conducted on two primary sets:
- Directions Set: 1,000 navigation queries containing French place names, synthesized using Parallel WaveNet to simulate rare OOV entities.
- General Traffic: 30,500 English utterances to monitor for regressions in standard performance.
Results and Discussion
The experimental results, summarized in Table 1, highlight the effectiveness of phoneme-based biasing. The phoneme-only biasing approach achieved a Word Error Rate (WER) of 27.5% on the Directions set, representing a 16% relative improvement over the grapheme baseline (32.8%) and an 8% improvement over the wordpiece baseline (30.0%). Combining wordpiece and phoneme biasing further reduced the WER to 26.9%.
A key finding is that phonemes are more acoustically salient than graphemes for OOV words. By focusing on how a word sounds rather than how it is spelled, the model becomes more robust to foreign entities. Furthermore, the model maintained high accuracy on general English tasks, with only a negligible 0.1% absolute WER increase.
However, the system's performance is sensitive to the size of the biasing list. As shown in Figure 1, the WER increases significantly as the number of biasing words grows from 1 to 1,000. This is primarily due to phonetic confusion between similar-sounding foreign names within the larger list.
Conclusion
This work demonstrates that a hybrid wordpiece-phoneme model can significantly improve the recognition of foreign named entities in English ASR. By leveraging phonemes as a universal acoustic representation and employing cross-lingual mapping, the model successfully recognizes OOV words that traditional grapheme models cannot. Future work will investigate the use of longer phonemic units to reduce phonetic confusion in large-scale biasing scenarios.
Summary updated