Introduction
Training high-quality Speech-to-Text (S2T) systems, encompassing automatic speech recognition (ASR) and speech translation (ST), typically necessitates massive centralized datasets. However, speech data is highly sensitive and subject to privacy regulations like GDPR. Federated Learning (FL) provides a decentralized alternative, training models on-device without data movement.
Despite its promise, standard FL algorithms like FedAvg face two significant hurdles in speech applications:
- Massive Communication Overhead: Large-scale S2T models (e.g., Transformers, Conformers) require prohibitive bandwidth to synchronize parameters between thousands of clients and a central server.
- Data Heterogeneity: Individual "speech profiles"—varying by accent, background noise, and hardware—create diverse data distributions that lead to parameter conflicts and degraded global model performance.
Proposed Methodology
To address these challenges, we propose a decoupled, two-stage framework combining efficient parameter tuning with local personalization.
Stage 1: FEDLORA for Communication Efficiency
Instead of updating the entire backbone model, we utilize Low-Rank Adaptation (LoRA). In this stage, the large backbone model (e.g., Whisper or Conformer) remains frozen. Only a small set of auxiliary rank decomposition matrices are trained and shared with the server. This approach drastically reduces the volume of data transmitted over the network and prevents "catastrophic forgetting" by preserving the model's pre-trained knowledge while adapting to federated data.
Stage 2: FEDMEM for Local Personalization
To handle diverse accents and domains without additional training, we introduce FEDMEM, a local memory-retrieval system based on a $k$-Nearest Neighbor ($k$NN) classifier. Each client constructs a local "datastore" of speech-representation-text triplets. During inference, the model queries this local memory to refine its neural predictions. This allows the system to adapt to a specific user's dialect or environment on the fly, effectively personalizing the global model with zero additional training overhead.
Experimental Setup and Datasets
The framework was evaluated on two major benchmarks representing real-world variability:
- CoVoST: Focused on dialect and accent diversity, featuring four English accents: England (EN), Australia (AU), India (IN), and Canada (CA).
- GigaSpeech: Focused on multi-domain diversity across ten categories (e.g., Gaming, News, Sports). This benchmark also included "invisible" clients to test generalization to new users.
- MuST-C: Utilized for the initial pre-training of the Conformer-based models.
Key Results
Communication and Accuracy Gains
As shown in the experimental results, the combination of FEDLORA and FEDMEM significantly outperforms standard federated approaches while requiring a fraction of the communication budget.
- Communication Savings: FEDLORA reduced communication costs by 91.4% to 96.5% compared to standard FedAvg. For instance, on the Whisper backbone, costs dropped from 112.71 GB to just 9.66 GB.
- Accuracy Improvements: On the CoVoST ASR task, the Whisper-based FEDLORA + FEDMEM approach achieved a lower average Word Error Rate (WER) of 14.20 compared to 14.88 for standard FedAvg. Similar improvements were observed in Speech Translation (ST) tasks, where the framework mitigated parameter conflicts to improve BLEU scores.
- Scalability and Generalization: In large-scale GigaSpeech tests, FEDLORA surpassed FedAvg by an average of 0.73 WER. Furthermore, new users joining the system can immediately benefit from the global model and quickly build their own FEDMEM datastores for rapid personalization.
Conclusion and Future Work
This two-stage framework demonstrates that efficient S2T federated learning is possible by decoupling global knowledge acquisition from local personalization. While the $k$NN lookup introduces a slight inference latency (maintaining ~92.9% of standard model speed), the massive gains in communication efficiency and local accuracy make it a viable path for privacy-preserving speech applications. Future research will explore scaling to even larger models and optimizing retrieval logic to further reduce latency.
Summary updated