TL;DR
In federated learning for hybrid speech recognition, combining language models from different decentralized clients is notoriously difficult because they use a mix of traditional n-gram matrices and neural networks. This paper introduces a novel "match-and-merge" paradigm that treats these structurally mismatched models as separate populations to optimize them jointly. Their reinforcement learning-based approach (RMMA) achieves near-centralized performance while converging up to seven times faster than a genetic algorithm baseline.
What Problem Does This Paper Address?
Federated learning (FL) allows multiple data curators to collaboratively train a global automatic speech recognition (ASR) model without directly sharing their private audio datasets. While merging the acoustic components of these models across clients is a well-studied problem, merging the language models (LMs) remains a major bottleneck for industrial hybrid ASR systems.
Hybrid ASR systems typically use a two-stage language modeling pipeline to balance speed and accuracy:
- A traditional, non-neural n-gram model to quickly generate a list of the top $N$ text hypotheses.
- A neural network (NN) model to rescore that list and pick the most accurate sentence.
This setup introduces two core challenges for federated learning:
- Heterogeneity: An n-gram model is essentially a frequency matrix, whereas an NN model consists of neural weights and biases. Because their architectures are fundamentally different, standard federated averaging (like FedAvg) cannot combine them.
- Alignment: The n-gram and NN models must be optimized in tandem. If you merge them independently, they lose their mutual alignment, leading to poor combined performance during the rescoring phase.
Methods
To bridge this structural gap, the authors propose a unified match-and-merge paradigm and design two distinct algorithms to execute it:
1. Genetic Match-and-Merge Algorithm (GMMA)
Inspired by natural selection, GMMA treats n-gram models and NN models as two separate populations. It applies type-specific genetic operations to evolve them over generations:
- For NN LMs: Crossover swaps layers between two models at a random point, and mutation randomly flips bits in the model's binary file.
- For n-gram LMs: Crossover performs a weighted average of two frequency matrices, and mutation scales randomly selected column vectors.
The algorithm pairs the top-$k$ performing n-gram models with the top-$k$ NN models based on their Character Error Rate (CER) on a validation set to breed the next generation.
2. Reinforced Match-and-Merge Algorithm (RMMA)
To overcome the slow, random nature of genetic algorithms, RMMA reformulates the merging task as a sequential decision-making problem guided by a reinforcement learning agent. Using an actor-critic framework, the agent learns to dynamically select the optimal blending weights and mutation perturbations for both model types. The agent receives a reward signal based directly on how much it reduces the CER on a validation dataset, allowing it to navigate the massive parameter search space with high efficiency.
Datasets
The authors evaluated their paradigm using seven publicly available Mandarin datasets from OpenSLR, totaling roughly 1,475.8 hours of broadcast, accent-diverse, and conversational speech:
- Source/Training Domains: SLR33 (Aishell), SLR38 (Free ST Mandarin), SLR47 (Primewords), SLR62 (aidatatang), and SLR93 (AISHELL-3) were used to train the initial local source models.
- Out-of-Domain Evaluation: SLR18 (THCHS-30) and SLR68 (Read Speech) were strictly reserved to test how well the final merged target models generalize to entirely unseen data distributions.
Key Insights and Ideas
- Co-Optimization is Mandatory: The paper demonstrates that language models in hybrid ASR cannot be treated as isolated components during federated aggregation. Jointly optimizing the n-gram and NN populations ensures they remain aligned for the N-best rescoring task.
- Targeted Rewards Beat Random Search: Genetic algorithms suffer from slow convergence because they rely on random mutations. By utilizing a CER-driven reward system, the RL agent in RMMA pinpoints high-quality model combinations significantly faster.
- Implicit Quality Weighting: The reinforcement learning agent naturally learns to assign higher weights to superior source models, effectively mitigating the negative impact of lower-quality or noisy local updates from poorer clients.
Key Results
- Near-Centralized Accuracy: RMMA achieved an average CER of 8.03% across the training domains, coming remarkably close to the centralized reference model (7.88%)—which represents the theoretical upper bound of training on all data simultaneously. It comfortably outperformed standard Direct Average (8.53%) and Fine-tuning (10.51%) baselines.
- Stronger Generalization: On the unseen SLR18 test set, RMMA achieved a CER of 13.68%, outperforming Direct Average (14.98%) and GMMA (14.08%), proving that the match-and-merge paradigm creates robust, generalizable models.
- Dramatic Speedups: RMMA converged in fewer than 30 iterations (under 2 days), whereas GMMA required over 800 iterations and approximately 15 days to reach a stable solution.
Limitations and Future Work
- Language Specificity: The experiments were conducted exclusively on Mandarin datasets. Mandarin features unique linguistic properties, such as character-level segmentation and high semantic ambiguity. The authors note that verifying the paradigm on alphabetic languages like English remains an open task.
- Scalability Constraints: The evaluation scaled up to 5 source models. In massive cross-device federated learning applications involving thousands of clients, the communication and computational overhead of running an RL agent sequentially across so many models requires further investigation.
Summary updated