Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition

Mengze Hong, Yi Gu, Di Jiang, Hanlin Gu, Chen Jason Zhang, Lu Wang, Zhiyang SuarXiv 2026 · arXiv

TL;DR

In federated learning for hybrid speech recognition, combining language models from different decentralized clients is notoriously difficult because they use a mix of traditional n-gram matrices and neural networks. This paper introduces a novel "match-and-merge" paradigm that treats these structurally mismatched models as separate populations to optimize them jointly. Their reinforcement learning-based approach (RMMA) achieves near-centralized performance while converging up to seven times faster than a genetic algorithm baseline.

What Problem Does This Paper Address?

Federated learning (FL) allows multiple data curators to collaboratively train a global automatic speech recognition (ASR) model without directly sharing their private audio datasets. While merging the acoustic components of these models across clients is a well-studied problem, merging the language models (LMs) remains a major bottleneck for industrial hybrid ASR systems.

Hybrid ASR systems typically use a two-stage language modeling pipeline to balance speed and accuracy:

  1. A traditional, non-neural n-gram model to quickly generate a list of the top $N$ text hypotheses.
  2. A neural network (NN) model to rescore that list and pick the most accurate sentence.

This setup introduces two core challenges for federated learning:

  • Heterogeneity: An n-gram model is essentially a frequency matrix, whereas an NN model consists of neural weights and biases. Because their architectures are fundamentally different, standard federated averaging (like FedAvg) cannot combine them.
  • Alignment: The n-gram and NN models must be optimized in tandem. If you merge them independently, they lose their mutual alignment, leading to poor combined performance during the rescoring phase.

Methods

To bridge this structural gap, the authors propose a unified match-and-merge paradigm and design two distinct algorithms to execute it:

1. Genetic Match-and-Merge Algorithm (GMMA)

Inspired by natural selection, GMMA treats n-gram models and NN models as two separate populations. It applies type-specific genetic operations to evolve them over generations:

  • For NN LMs: Crossover swaps layers between two models at a random point, and mutation randomly flips bits in the model's binary file.
  • For n-gram LMs: Crossover performs a weighted average of two frequency matrices, and mutation scales randomly selected column vectors.

The algorithm pairs the top-$k$ performing n-gram models with the top-$k$ NN models based on their Character Error Rate (CER) on a validation set to breed the next generation.

Fig. 2. Overview of the Genetic Match-and-Merge Algorithm (GMMA). Heterogeneous language models (n-gram and neural LMs) are treated as separate populations and evolved via type-specific genetic operations. The top-k candidates from each population are then paired, and LM combinations with the highest matching fitness are selected for reproduction and merging.

2. Reinforced Match-and-Merge Algorithm (RMMA)

To overcome the slow, random nature of genetic algorithms, RMMA reformulates the merging task as a sequential decision-making problem guided by a reinforcement learning agent. Using an actor-critic framework, the agent learns to dynamically select the optimal blending weights and mutation perturbations for both model types. The agent receives a reward signal based directly on how much it reduces the CER on a validation dataset, allowing it to navigate the massive parameter search space with high efficiency.

Fig. 3. Overview of Reinforced Match-and-Merge Algorithm. The merging problem is formulated as a sequential decision-making pro- cess, where a reinforcement learning agent selects actions to maxi- mize rewards and enhance model quality.

Datasets

The authors evaluated their paradigm using seven publicly available Mandarin datasets from OpenSLR, totaling roughly 1,475.8 hours of broadcast, accent-diverse, and conversational speech:

  • Source/Training Domains: SLR33 (Aishell), SLR38 (Free ST Mandarin), SLR47 (Primewords), SLR62 (aidatatang), and SLR93 (AISHELL-3) were used to train the initial local source models.
  • Out-of-Domain Evaluation: SLR18 (THCHS-30) and SLR68 (Read Speech) were strictly reserved to test how well the final merged target models generalize to entirely unseen data distributions.

Key Insights and Ideas

  • Co-Optimization is Mandatory: The paper demonstrates that language models in hybrid ASR cannot be treated as isolated components during federated aggregation. Jointly optimizing the n-gram and NN populations ensures they remain aligned for the N-best rescoring task.
  • Targeted Rewards Beat Random Search: Genetic algorithms suffer from slow convergence because they rely on random mutations. By utilizing a CER-driven reward system, the RL agent in RMMA pinpoints high-quality model combinations significantly faster.
  • Implicit Quality Weighting: The reinforcement learning agent naturally learns to assign higher weights to superior source models, effectively mitigating the negative impact of lower-quality or noisy local updates from poorer clients.

Key Results

  • Near-Centralized Accuracy: RMMA achieved an average CER of 8.03% across the training domains, coming remarkably close to the centralized reference model (7.88%)—which represents the theoretical upper bound of training on all data simultaneously. It comfortably outperformed standard Direct Average (8.53%) and Fine-tuning (10.51%) baselines.
  • Stronger Generalization: On the unseen SLR18 test set, RMMA achieved a CER of 13.68%, outperforming Direct Average (14.98%) and GMMA (14.08%), proving that the match-and-merge paradigm creates robust, generalizable models.
  • Dramatic Speedups: RMMA converged in fewer than 30 iterations (under 2 days), whereas GMMA required over 800 iterations and approximately 15 days to reach a stable solution.
Fig. 4. Convergence behavior on training data. The secondary axis (right) corresponds to GMMA due to its larger variance.

Limitations and Future Work

  • Language Specificity: The experiments were conducted exclusively on Mandarin datasets. Mandarin features unique linguistic properties, such as character-level segmentation and high semantic ambiguity. The authors note that verifying the paradigm on alphabetic languages like English remains an open task.
  • Scalability Constraints: The evaluation scaled up to 5 source models. In massive cross-device federated learning applications involving thousands of clients, the communication and computational overhead of running an RL agent sequentially across so many models requires further investigation.

Summary updated

Abstract

Training automatic speech recognition (ASR) models increasingly relies on decentralized federated learning to ensure data privacy and accessibility, producing multiple local models that require effective merging. In hybrid ASR systems, while acoustic models can be merged using established methods, the language model (LM) for rescoring the N-best speech recognition list faces challenges due to the heterogeneity of non-neural n-gram models and neural network models. This paper proposes a heterogeneous LM optimization task and introduces a match-and-merge paradigm with two algorithms: the Genetic Match-and-Merge Algorithm (GMMA), using genetic operations to evolve and pair LMs, and the Reinforced Match-and-Merge Algorithm (RMMA), leveraging reinforcement learning for efficient convergence. Experiments on seven OpenSLR datasets show RMMA achieves the lowest average Character Error Rate and better generalization than baselines, converging up to seven times faster than GMMA, highlighting the paradigm's potential for scalable, privacy-preserving ASR systems.
BibTeX
@misc{hong2026federated,
  author = {Mengze Hong and Yi Gu and Di Jiang and Hanlin Gu and Chen Jason Zhang and Lu Wang and Zhiyang Su},
  title = {{Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition}},
  year = {2026},
  eprint = {2603.04945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.04945},
}
Figure