Abstract
Training large-scale Automatic Speech Recognition (ASR) models often faces challenges when data is siloed on user devices with highly heterogeneous characteristics. This paper introduces Dynamic Gradient Aggregation (DGA), a federated learning algorithm that utilizes hierarchical optimization and adaptive gradient weighting to mitigate the impact of non-i.i.d. data. Experimental results on the LibriSpeech benchmark demonstrate that DGA achieves a significant speedup in convergence and superior accuracy compared to traditional Federated Averaging (FedAvg), reaching a Word Error Rate (WER) of 4.40%.
Introduction
Distributed training typically assumes independent and identically distributed (i.i.d.) data across nodes. However, in real-world Federated Learning (FL) scenarios—particularly for privacy-sensitive applications like speech recognition—data is often non-i.i.d. due to varying acoustic environments, accents, and hardware. Standard algorithms like FedAvg struggle with this heterogeneity, often resulting in slow convergence or suboptimal performance. This work proposes DGA to bridge this gap by creating a robust aggregation scheme that handles outlier data without compromising the global model.
Methodology
The proposed approach consists of two primary components:
1. Hierarchical Optimization
Instead of a simple one-step update, a two-level optimization scheme is employed. Clients perform local updates using their own optimizers, while the server utilizes a "global" optimizer to aggregate these updates. This structure significantly enhances convergence speed.
2. Dynamic Gradient Aggregation (DGA)
DGA assigns non-uniform weights to client updates to account for data quality and relevance. Two variants are explored:
- $DGA_{SM}$ (Softmax): Weights are derived from training losses, de-emphasizing clients with higher losses that may represent poorly modeled or outlier data.
- $DGA_{RL}$ (Reinforcement Learning): A separate neural network is trained via RL to infer optimal aggregation weights, rewarded based on the global model's performance on a server-side validation set.
Additionally, the framework incorporates an unsupervised adaptation pipeline using Text-to-Speech (TTS) for synthetic data generation and a "Naive Rehearsal" strategy to mitigate catastrophic forgetting.
Experimental Evaluation
The methods were evaluated across three scenarios: LibriSpeech (supervised ASR), Internal Meeting/Presentation Data (unsupervised session adaptation), and Conversational Speech (semi-supervised).
LibriSpeech Results
On the LibriSpeech task, DGA variants significantly outperformed the FedAvg baseline. While FedAvg required approximately 800 iterations to converge, $DGA_{RL}$ reached convergence in only 144 iterations, representing a 7x speedup.
As shown in the table below, $DGA_{RL}$ achieved the lowest WER of 4.40%, surpassing both the centralized online baseline and standard FL approaches.
Discussion and Conclusion
The DGA algorithm effectively acts as a regularizer by de-emphasizing gradients that deviate from the consensus, thereby reducing the variance of the aggregated gradient. This theoretical property translates to the observed empirical gains in convergence speed and accuracy. While the current focus is on ASR, the principles of DGA are potentially applicable to other domains such as computer vision and NLP. Future work will investigate the scalability of $DGA_{RL}$ in settings where server-side validation data is limited.
Summary updated