TL;DR
This paper introduces Federated Marginal Personalization (FMP), a novel approach to personalize Automatic Speech Recognition (ASR) language models on user devices using federated learning. FMP adapts NNLMs by adjusting word probabilities based on estimated global and personalized word distributions, overcoming limitations of traditional federated fine-tuning and offering a privacy-preserving solution.
What Problem Does This Paper Address?
Modern ASR systems often struggle to adapt to individual users' unique speaking styles, vocabulary, and domain-specific language. While federated learning (FL) offers a way to train models on private user data without it leaving the device, directly fine-tuning large neural network language models (NNLMs) on limited personal data can lead to overfitting or catastrophic forgetting. Furthermore, users might not generate enough data to effectively update the model. This paper addresses the challenge of efficiently and effectively personalizing NNLMs on user devices within a federated learning framework, ensuring both accuracy and privacy.
Methods
The core of the proposed method is Federated Marginal Personalization (FMP). Instead of fine-tuning the entire NNLM, FMP focuses on adapting the model's output probabilities. This is achieved by:
- Estimating Marginal Distributions: FMP continuously estimates two types of unigram (single-word) distributions:
- Global Unigram Distribution ($\bar{q}_t(w)$): Aggregated from all participating clients at each federated learning round.
- Personalized Unigram Distribution ($q_i^t(w)$): Estimated from the user's historical ASR decoded text on their local device.
- Marginal Adaptation Factor: An adaptation factor, $g(w)$, is computed by interpolating the background unigram distribution ($u(w)$), the global unigram distribution ($\bar{q}_t(w)$), and the personalized unigram distribution ($q_i^t(w)$). This factor is designed to balance general language patterns, trending words, and user-specific vocabulary.
- Probability Adjustment: The original NNLM probabilities ($P_{nnlm}(w|h)$) are then adjusted by multiplying them with this adaptation factor, effectively creating a personalized NNLM ($P_{i,t+1}^{nnlm}(w|h)$).
- Federated Averaging for Global Distribution: The personalized unigram distributions from clients are sent to the server and aggregated using a federated averaging approach to update the global unigram distribution for the next round.
- Privacy Mechanism: To ensure differential privacy, a Laplace mechanism is applied to the server-side update of the global unigram distribution, adding noise proportional to the sensitivity of the query.
Datasets
The paper evaluates FMP on two speech evaluation datasets:
- Curated Video: A carefully selected set of clean videos, segmented into utterances.
- Augmented Multi-Party Interaction (AMI) Meeting data: This dataset includes both scenario and non-scenario meetings, with participants having assigned roles or free to converse.
For training the general background NNLM, a corpus of approximately 30 million English sentences from public Facebook posts and comments was used.
Key Insights and Ideas
- Personalization without Full Fine-tuning: The key insight is that adapting the output probabilities based on marginal word distributions can achieve personalization more efficiently and with fewer privacy risks than full model fine-tuning.
- Balancing Global and Local Information: The interpolation of background, global, and personalized unigram distributions allows the model to leverage diverse linguistic information, preventing overfitting to limited personal data and incorporating trending or domain-specific language.
- Leveraging First-Pass Decodings: The method cleverly uses the output of the first-pass ASR decoder (N-best hypotheses) to estimate personalized unigram distributions, making it an unsupervised personalization approach.
- Differential Privacy for Unigram Distributions: Applying differential privacy to the aggregation of unigram distributions provides a strong privacy guarantee at the utterance level, with a manageable impact on accuracy.
Key Results
- FMP consistently improved Word Error Rate (WER) on both Curated Video (2.4% relative gain) and AMI Meeting (4.8% relative gain) datasets compared to the baseline (first-pass ASR with general NNLM rescoring).
- The performance improvement generally increased with more federated learning rounds (T), indicating that more frequent updates lead to better personalization.
- The choice of hyperparameters, particularly the interpolation weights ($\alpha$, $\beta$) and the Gaussian kernel bandwidth ($\sigma$) for estimating personalized unigrams, significantly impacts performance. $\sigma=1$ was found to be optimal for balancing the use of N-best hypotheses.
- Randomized FMP with differential privacy achieved reasonable utterance-level privacy protection with a relatively small cost in speech recognition accuracy. For $\epsilon \ge 0.5$, the WER degradation was less than 1%.
Limitations and Future Work
- Utterance-Level Privacy: The paper acknowledges that while utterance-level differential privacy is achieved, a more robust definition for ASR tasks would be user-level adjacency, which protects entire user histories. This is left for future work.
- Higher-Order N-grams: The current approach focuses on unigram distributions. The authors suggest exploring higher-order n-gram distributions for deeper personalization in future research.
- Computational Cost: While FMP is more efficient than full fine-tuning, the continuous estimation of unigram distributions and the federated averaging process still incur computational costs on user devices.
- Soft Labels: The paper mentions an alternative approach of using soft labels predicted by ASR models for unigram estimation, which was not explored in this study.
Summary updated