TL;DR
This paper introduces Pronunciation-aware Embeddings (PET) for Transducer-based Automatic Speech Recognition (ASR) models. By sharing embedding components for text tokens with similar pronunciations, PET improves accuracy and, crucially, mitigates "error chain reactions" where one mistake leads to a cascade of subsequent errors.
What Problem Does This Paper Address?
Traditional end-to-end ASR models, while powerful, often treat each text token as an independent entity. This overlooks linguistic nuances like homophones (words that sound alike but have different meanings and spellings), which are prevalent in languages like Mandarin Chinese. Conventional Transducers, a popular end-to-end architecture, independently train embeddings for each token. This can lead to suboptimal performance, especially when dealing with homophones, as the model doesn't leverage their shared pronunciation. Furthermore, the paper identifies a phenomenon called "error chain reactions," where recognition errors tend to cluster, with one error increasing the likelihood of subsequent errors. This makes it harder to improve ASR performance beyond a certain point.
Methods
The core innovation is the Pronunciation-aware Embeddings (PET) approach for the Transducer decoder. Instead of a single, independent embedding for each text token, PET constructs embeddings by summing contributions from various features. These features can include the token's identity (like a traditional embedding) and, crucially, its pronunciation. The paper proposes using features derived from the Romanized pronunciation, including its initial consonant(s) (C) and its suffix (V), and optionally tone (T) for tonal languages.
By combining these features, tokens with the same or similar pronunciations can share embedding components. For instance, two Mandarin characters pronounced "ta" would share the pronunciation embedding component, even if their character IDs are different. This allows the model to leverage phonetic similarities. The paper also explores incorporating pronunciation information into the "joiner" network, which combines encoder and decoder outputs, though this proved less effective than embedding-level pronunciation awareness.
Datasets
The proposed PET models were evaluated on:
- Mandarin Chinese:
- AISHELL-2: A large-scale dataset with 1000 hours of speech.
- THCHS: A smaller, out-of-domain dataset.
- Korean:
- Zeroth-Korean: A dataset with approximately 50 hours of speech.
Key Insights and Ideas
- Homophones are a missed opportunity: Treating homophones as entirely independent tokens in ASR is suboptimal. Leveraging their shared pronunciation can lead to performance gains.
- Error Chain Reactions are a bottleneck: ASR errors are not randomly distributed; they tend to cluster. This "error chain reaction" phenomenon significantly hinders further accuracy improvements.
- PET tackles both issues: By using pronunciation-aware embeddings, PET not only improves accuracy but also directly addresses error chain reactions. When an error occurs, if the pronunciation is still somewhat captured by the embedding, the decoder might still produce a "correct" embedding, preventing the error from cascading.
- Granularity matters: More granular pronunciation features (like consonant-vowel splits) seem to yield better results than broader representations.
- Pronunciation in embeddings is key, not just the joiner: While incorporating pronunciation into the joiner was explored, the most significant gains came from modifying the decoder embeddings.
Key Results
- Consistent Accuracy Improvement: PET models consistently improved Character Error Rate (CER) on both Mandarin Chinese and Korean datasets compared to baseline Transducers. For example, on the THCHS dataset, PET achieved an absolute reduction of 1.01% CER.
- Mitigation of Error Chain Reactions: Table 6 shows a significant reduction in the probability of an error occurring given a previous error (P(EE)) when using PET models, especially when word identity is excluded from the decoder embeddings. This indicates PET effectively breaks error chains.
- Reduced Average Error Cluster Length: Table 7 demonstrates that models using only pronunciation for decoder embeddings have shorter average error cluster lengths, further supporting the mitigation of error chain reactions.
- Model Size: The paper notes that while PET might seem to increase parameter count due to multiple embedding tables, the final embedding table can be pre-computed, resulting in the same model size and inference speed as conventional Transducers.
Limitations and Future Work
- Language Dependence: The effectiveness of incorporating pronunciation into the joiner embedding varied between Mandarin Chinese and Korean. The authors hypothesize this is due to data size differences, with smaller datasets benefiting more from joint pronunciation information. Further investigation is needed.
- Error Chain Reaction Analysis: While the paper identifies and quantifies error chain reactions, further research is suggested to fully understand and exploit this phenomenon.
- Open-sourcing: The authors plan to open-source their implementation with the NeMo toolkit, which will facilitate further research and adoption.
Summary updated