Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

Yichao Du, Zhirui Zhang, Linan Yue, Xu Huang, Yuqing Zhang, Tong Xu, Linli Xu, Enhong Chen

Introduction

Training high-quality Speech-to-Text (S2T) systems, encompassing automatic speech recognition (ASR) and speech translation (ST), typically necessitates massive centralized datasets. However, speech data is highly sensitive and subject to privacy regulations like GDPR. Federated Learning (FL) provides a decentralized alternative, training models on-device without data movement.

Despite its promise, standard FL algorithms like FedAvg face two significant hurdles in speech applications:

  1. Massive Communication Overhead: Large-scale S2T models (e.g., Transformers, Conformers) require prohibitive bandwidth to synchronize parameters between thousands of clients and a central server.
  2. Data Heterogeneity: Individual "speech profiles"—varying by accent, background noise, and hardware—create diverse data distributions that lead to parameter conflicts and degraded global model performance.

Proposed Methodology

To address these challenges, we propose a decoupled, two-stage framework combining efficient parameter tuning with local personalization.

Overview of the FEDLORA and FEDMEM framework, illustrating the two-stage process of lightweight parameter updates and local memory retrieval.

Stage 1: FEDLORA for Communication Efficiency

Instead of updating the entire backbone model, we utilize Low-Rank Adaptation (LoRA). In this stage, the large backbone model (e.g., Whisper or Conformer) remains frozen. Only a small set of auxiliary rank decomposition matrices are trained and shared with the server. This approach drastically reduces the volume of data transmitted over the network and prevents "catastrophic forgetting" by preserving the model's pre-trained knowledge while adapting to federated data.

Stage 2: FEDMEM for Local Personalization

To handle diverse accents and domains without additional training, we introduce FEDMEM, a local memory-retrieval system based on a $k$-Nearest Neighbor ($k$NN) classifier. Each client constructs a local "datastore" of speech-representation-text triplets. During inference, the model queries this local memory to refine its neural predictions. This allows the system to adapt to a specific user's dialect or environment on the fly, effectively personalizing the global model with zero additional training overhead.

Experimental Setup and Datasets

The framework was evaluated on two major benchmarks representing real-world variability:

  • CoVoST: Focused on dialect and accent diversity, featuring four English accents: England (EN), Australia (AU), India (IN), and Canada (CA).
  • GigaSpeech: Focused on multi-domain diversity across ten categories (e.g., Gaming, News, Sports). This benchmark also included "invisible" clients to test generalization to new users.
  • MuST-C: Utilized for the initial pre-training of the Conformer-based models.

Key Results

Communication and Accuracy Gains

As shown in the experimental results, the combination of FEDLORA and FEDMEM significantly outperforms standard federated approaches while requiring a fraction of the communication budget.

Performance comparison and communication costs on the CoVoST benchmark, highlighting the efficiency of FEDLORA and the accuracy gains from FEDMEM.
  • Communication Savings: FEDLORA reduced communication costs by 91.4% to 96.5% compared to standard FedAvg. For instance, on the Whisper backbone, costs dropped from 112.71 GB to just 9.66 GB.
  • Accuracy Improvements: On the CoVoST ASR task, the Whisper-based FEDLORA + FEDMEM approach achieved a lower average Word Error Rate (WER) of 14.20 compared to 14.88 for standard FedAvg. Similar improvements were observed in Speech Translation (ST) tasks, where the framework mitigated parameter conflicts to improve BLEU scores.
  • Scalability and Generalization: In large-scale GigaSpeech tests, FEDLORA surpassed FedAvg by an average of 0.73 WER. Furthermore, new users joining the system can immediately benefit from the global model and quickly build their own FEDMEM datastores for rapid personalization.

Conclusion and Future Work

This two-stage framework demonstrates that efficient S2T federated learning is possible by decoupling global knowledge acquisition from local personalization. While the $k$NN lookup introduces a slight inference latency (maintaining ~92.9% of standard model speed), the massive gains in communication efficiency and local accuracy make it a viable path for privacy-preserving speech applications. Future research will explore scaling to even larger models and optimizing retrieval logic to further reduce latency.

Summary updated

BibTeX
@inproceedings{du2024communication,
  author = {Yichao Du and Zhirui Zhang and Linan Yue and Xu Huang and Yuqing Zhang and Tong Xu and Linli Xu and Enhong Chen},
  title = {{Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447662},
}
Figure