TL;DR
This paper introduces "phone mapping," a novel technique for training acoustic models that can understand multiple English dialects simultaneously. By mapping phones from different dialects to a common set, they achieve significant improvements in speech recognition accuracy compared to traditional methods, while also simplifying model training and maintenance.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems often struggle when faced with variations in how people speak, such as different accents or dialects. Training separate models for each dialect is resource-intensive and doesn't scale well, especially for less common dialects. Existing methods for unifying these models, like sharing hidden layers (SHL) or using transfer learning, have limitations. SHL training can be slow and doesn't fully share model parameters, while transfer learning is less effective for combining many dialects. The challenge is to create a single acoustic model that performs well across multiple dialects, is efficient to train, and easy to deploy and adapt.
Methods
The core contribution of this paper is the phone mapping technique. The idea is to select one dialect's phone set and acoustic tree as "canonical" and then map the phones from all other dialects to this canonical set. This allows for a unified acoustic model where all network parameters can be shared.
The authors compare phone mapping against:
- Transfer Learning: Training a model on one dialect and then adapting it to another.
- Shared Hidden Layer (SHL) Training: A popular method where the hidden layers of a neural network are shared across dialects, but the output layers remain separate.
- One-Hot Dialect Vectors: Auxiliary input features that explicitly tell the model which dialect it's processing.
- Frame-Level Online i-Vectors: Features used for adapting acoustic models on-the-fly, crucial for real-time applications like voice assistants.
The experiments use an LSTM-HMM framework. For phone mapping, they map non-canonical phones to their nearest canonical counterparts, a process that requires linguistic input but is done once. They also explore combining phone mapping with one-hot dialect vectors and i-vectors.
Datasets
The experiments were conducted using datasets comprising four English dialects:
- US (American)
- GB (British)
- IN (Indian)
- AU (Australian)
Two primary training dataset sizes were used:
- 2K hour dataset: Used for initial experiments and comparisons.
- 40K hour dataset: Used to evaluate the scalability and robustness of phone mapping with larger data.
The test data was common to both setups and consisted of utterances collected from far-field devices, with no overlap in device units or speakers between train and test sets. For the 40K hour dataset, the training data included corrupted copies of original recordings generated via simulated reverberation.
Key Insights and Ideas
- Phone mapping is a powerful unification strategy: It offers significant improvements over dialect-specific models and is simpler to implement and maintain than SHL training.
- Unified models benefit all dialects, even dominant ones: Contrary to some prior assumptions, pooling data and using phone mapping improved performance even for the American English dialect, which contributed the most data.
- i-vectors provide a consistent boost: Frame-level i-vectors consistently improved accuracy when used with both phone mapping and SHL training, offering additional gains.
- One-hot dialect vectors have limited impact: Surprisingly, one-hot dialect vectors showed little to no benefit, even when combined with i-vectors. This contradicts some previous findings and suggests that phone mapping might implicitly handle dialectal variations effectively.
- Sequence discriminative training amplifies gains: The benefits of phone mapping become even more pronounced after sequence discriminative training (sMBR), indicating that the cross-entropy stage model is somewhat coarse and sMBR helps correct phone confusions.
- Canonical tree is better than mixed tree: A model built with a canonical tree (phone mapping) outperformed a model with a mixed tree (where the acoustic tree is estimated by pooling data from all dialects), suggesting that it's better to capture intra-dialect variations and map cross-dialect differences.
Key Results
- 2K hour dataset:
- Phone mapping achieved average Word Error Rate (WER) reductions of 14% over dialect-specific training and 5% over transfer learning.
- SHL training was only slightly better than phone mapping.
- i-vectors provided an additional 3% average WER reduction, while one-hot encoding had little effect.
- 40K hour dataset:
- Phone mapping showed healthy WER reductions of 10% on average over dialect-specific models, even after sequence discriminative training.
- Gains after sMBR training were larger than after cross-entropy training.
Limitations and Future Work
- Linguistic Overhead for Phone Mapping: While minimal, phone mapping requires linguistic input to define phone mappings. The paper notes that this overhead is manageable as mappings are determined once.
- One-Hot Vector Ineffectiveness: The paper highlights that one-hot dialect vectors had little effect, which is contrary to some prior work. The authors hypothesize about why this might be the case but suggest further investigation.
- Future Work: The paper implicitly suggests that further exploration into why one-hot vectors are not effective in their setup could be valuable. Additionally, the simplicity and effectiveness of phone mapping make it a strong candidate for future research in multi-dialect and multilingual ASR. The paper also notes that phone mapping simplifies online adaptation via HMM-based i-vectors by allowing a single T-matrix for all dialects, hinting at its practical deployment advantages.
Summary updated