TL;DR
This paper introduces a novel method for personalizing speech recognition systems on mobile devices by adapting to individual users and their environments. They achieve this by generalizing the i-vector framework to cluster utterances and learn device-specific feature transformations, leading to significant reductions in word error rates.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems, especially on mobile devices, face a significant challenge: acoustic variability. This variability stems from different speakers using the device, as well as diverse acoustic environments (e.g., a car, a street, a quiet room). Standard ASR models trained on general data struggle to perform well under these conditions. Personalization aims to bridge this gap by adapting the ASR system to a specific user and their typical usage environment. While speaker adaptation and feature transformation techniques exist, this paper focuses on a more robust and generalized approach that can capture both speaker and environment variations within a specific device context.
Methods
The core of this paper is a generalized i-vector clustering method for personalization. Here's a breakdown of their approach:
- Generalized i-Vector Extraction: They extend the standard i-vector model (which represents an utterance's deviation from a universal background model) by introducing:
- Tied Sub-Loading Matrices: Instead of each Gaussian component in the Universal Background Model (UBM) having its own unique loading matrix, they group components into "regression classes" and share loading matrices within these classes. This reduces the number of parameters and can improve robustness.
- Residual Noise Modeling: They explicitly model residual noise, which accounts for inaccuracies in the i-vector representation.
- Device-Specific UBM Training: Crucially, they train the UBM only on data from the specific device being personalized. This ensures the i-vectors capture variability relevant to that device's typical usage.
- Clustering and Feature Transformation:
- They train a GMM on the extracted i-vectors from the device's training data.
- At test time, an utterance's i-vector is extracted, and then assigned to the closest cluster centroid in the GMM.
- A Constrained Maximum Likelihood Linear Regression (CMLLR) feature transform is then applied, tailored to the specific cluster the utterance belongs to. This cluster-specific transform compensates for the identified speaker/environment variability.
Datasets
- Internal Large Vocabulary Voice Search System for Gaming: Used for recognition experiments. This dataset consists of 20 hours of Xbox voice search commands for training and 2 hours for testing. Each device in both training and testing had 70 utterances.
- Single Speaker Data: Used for initial experiments on the impact of i-vector parameters on log-likelihood scores and clustering. This involved 70 utterances from one speaker.
Key Insights and Ideas
- Device-Centric Personalization is Key: Training the UBM specifically on data from a single device is a critical step. This allows the i-vectors to effectively capture the unique variability patterns encountered on that particular device, rather than general acoustic variations.
- Tying and Residual Noise Improve Robustness: While introducing residual noise modeling had a modest impact on log-likelihood, tying sub-loading matrices (even with a reduction in likelihood) was shown to be beneficial in the overall personalization framework, likely by regularizing the model.
- Clustering Unlocks Targeted Adaptation: The ability to cluster utterances and apply cluster-specific feature transformations is more effective than a single, device-wide adaptation. This allows for finer-grained compensation of different acoustic conditions or speaker characteristics that might co-exist on a single device.
- Optimal Cluster Number Varies: The optimal number of clusters for personalization isn't fixed. It appears to depend on the underlying acoustic modeling technique (MLE, MMI, bMMI), with more robust models potentially requiring fewer clusters.
Key Results
The paper reports significant improvements in word error rates (WER) for their personalization method:
- 28% Reduction in WER: Compared to a per-utterance adaptation system on their internal large vocabulary voice search task.
- Hierarchical Improvement:
- "Per device, 1 cluster" (device-wide adaptation) reduced WER by 15-17% compared to "per utterance" adaptation.
- Further splitting into 2 or 4 clusters ("per dev 2cls", "per dev 4cls") led to additional reductions of 7-12% and 4-10% respectively, depending on the acoustic model (MLE, MMI, bMMI).
- Optimal Clusters: For MLE models, 4 clusters yielded the best results. For MMI and bMMI systems, 2 clusters were sufficient.
Limitations and Future Work
- Limitations:
- The paper notes that introducing residual noise had a relatively small impact on log-likelihood scores in their initial experiments.
- The optimal number of clusters was found to be dependent on the acoustic modeling approach, suggesting a need for methods to automatically determine this.
- Future Work:
- The authors plan to extend their method to subspace fMLLR, which could potentially train cluster-specific basis transformations. This could offer even more powerful adaptation capabilities.
Summary updated