Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)

Jae Hun Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Theertha Suresh, Shankar Kumar, Rajiv Mathews Google, Xinwei Wu, Li Gong, Deyi Xiong, Salman Avestimehr, Melas-Kyriazi Luke, Jae Hun, Anit Kumar Sahu, Eric W. Tramel, Mathews, Lara, Theertha, William, Wang, Deyi, Zhang, Enmao Diao, Jie Ding, Vahid Tarokh, Alec Go, Richa Bhayani, Lei Huang, Qiuyi Han, M Edoardo, Vahid Airoldi, Tian Li, Shengyuan Hu, Ahmad Beirami, Virginia Smith, Wei Yang, Bryan Lim, Cong Nguyen, Dinh Thai Luong, Yutao Hoang, Ying-Chang Jiao, Qiang Liang, Dusit Yang, Chunyan Niyato, Miao, Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas, Amanda Purington, Jessie Taft, Shruti Sannon, Natalya Bazarova, Samuel Hardman, Taylor, Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, Milan Vojnovic, Jeremy Bernstein, Kamyar Azizzadenesheli, Animashree Anandkumar, Theodora Brisimi, Ruidi Chen, Theofanie Mela, Alex Olshevsky, Ioannis Ch Paschalidis, Wei Shi, Adeline Wong, Cyril Allauzen, Franoise Beaufays, Michael Riley, Valerii Krzysztof Marcin Choromanski, David Likhosherstov, Xingyou Dohan, Andreea Song, Tamas Gane, Peter Sarlos, Jared Hawkins, Afroz Davis, Lukasz Mohiuddin, David Kaiser, Lucy BelangerOpenalex Unknown 2022 · DOI · OpenAlex

Abstract

In the context of personalized federated learning (FL), the critical challenge is to balance local model improvement and global model tuning when the personal and global objectives may not be exactly aligned. Inspired by Bayesian hierarchical models, we develop Act-PerFL, a self-aware personalized FL method where each client can automatically balance the training of its local personal model and the global model that implicitly contributes to other clients' training. Such a balance is derived from the inter-client and intra-client uncertainty quantification. Consequently, Act-PerFL can adapt to the underlying clients' heterogeneity with uncertainty-driven local training and model aggregation. With experimental studies on Sent140 and Amazon Alexa audio data, we show that Act-PerFL can achieve superior personalization performance compared with the existing counterparts.

Introduction

Federated Learning (FL) has emerged as a pivotal paradigm for training machine learning models on decentralized data while preserving user privacy. However, the inherent heterogeneity of edge devices—ranging from varying user habits on smartphones to diverse acoustic environments for smart speakers—presents a significant challenge. A standard global model often fails to capture individual nuances, leading to the "personalization gap."

Personalized FL aims to bridge this gap by tailoring models to individual users. The core difficulty lies in balancing the "tug-of-war" between local fine-tuning and global knowledge retention. Excessive local training leads to overfitting on small datasets, while insufficient training results in a generic model that lacks personal relevance. Current approaches often rely on manual hyperparameter tuning, which is unscalable for millions of diverse users.

Collaborating organizations for the Act-PerFL research, including FedML, Intel Labs, and Amazon Alexa.

Methodology: The Act-PerFL Framework

Presented at the FL4NLP 2022 workshop, Act-PerFL (Active Personalized Federated Learning) addresses personalization through the lens of statistical uncertainty. By employing a Bayesian Hierarchical Model, the framework treats the relationship between users as a probabilistic hierarchy.

Uncertainty Quantification

Act-PerFL models two distinct layers of uncertainty:

  1. Intra-client uncertainty: Captures the variance and noise within a single user's local dataset.
  2. Inter-client uncertainty: Measures the degree of divergence between different users across the network.

Self-Aware Local Training

By quantifying these uncertainties in real-time, Act-PerFL enables clients to become "self-aware." The system automatically derives:

  • Optimal Initialization: A customized starting point for each training round based on the global prior.
  • Adaptive Local Steps: A mathematically derived number of local training iterations (early stopping) to prevent overfitting.
  • Uncertainty-Weighted Aggregation: A server-side mechanism that weights client updates based on quantified certainty rather than simple sample counts.

Experimental Evaluation

The researchers evaluated Act-PerFL across two heterogeneous datasets representing distinct modalities:

Amazon Alexa Audio Data

This large-scale, real-world dataset for wake-word detection comprises 39,000 hours of training data. It spans five device types (e.g., Echo Dot, Echo Show), each characterized by unique acoustic profiles and hardware constraints.

Sentiment140

A text-based sentiment analysis dataset involving 772 clients, used to simulate non-IID (Independent and Identically Distributed) user behaviors in natural language processing tasks.

Key Results and Insights

  • Superior Audio Performance: On the Alexa dataset, Act-PerFL achieved the lowest relative false accept (FA) rates. The global model reached relative FA scores of 0.91–0.94, outperforming standard FedAvg and specialized methods like DITTO. Personalized models achieved scores as low as 0.90.
  • Faster Convergence in NLP: On the Sent140 dataset, Act-PerFL demonstrated significantly faster convergence in both training and test accuracy compared to state-of-the-art methods like pFedMe, requiring fewer communication rounds to reach peak performance.
  • Deep Learning Compatibility: Although rooted in Gaussian statistics, the authors implemented Act-PerFL for complex architectures (LSTMs and CNNs) using recursive online variance estimation, ensuring minimal computational overhead.

Conclusion

Act-PerFL frames personalization as a variance reduction problem, allowing clients to autonomously balance local and global objectives. By moving away from fixed heuristics toward uncertainty-driven adaptation, the framework provides a robust solution for large-scale, heterogeneous federated learning environments. Future research will focus on extending the theoretical framework to non-convex loss landscapes and improving stability for extremely sparse local datasets.

Summary updated

Abstract

In the context of personalized federated learning (FL), the critical challenge is to balance local model improvement and global model tuning when the personal and global objectives may not be exactly aligned. Inspired by Bayesian hierarchical models, we develop Act-PerFL, a self-aware personalized FL method where each client can automatically balance the training of its local personal model and the global model that implicitly contributes to other clients' training. Such a balance is derived from the inter-client and intra-client uncertainty quantification. Consequently, Act-PerFL can adapt to the underlying clients' heterogeneity with uncertainty-driven local training and model aggregation. With experimental studies on Sent140 and Amazon Alexa audio data, we show that ActPerFL can achieve superior personalization performance compared with the existing counterparts.
BibTeX
@inproceedings{ro2022proceedings,
  author = {Jae Hun Ro and Theresa Breiner and Lara McConnaughey and Mingqing Chen and Ananda Theertha Suresh and Shankar Kumar and Rajiv Mathews Google and Xinwei Wu and Li Gong and Deyi Xiong and Salman Avestimehr and Melas-Kyriazi Luke and Jae Hun and Anit Kumar Sahu and Eric W. Tramel and Gong and Kumar and Mathews and Mcconnaughey and Lara and Melas-Kyriazi Luke and Jae Hun and Anit Sahu and Theertha and Eric Tramel and William and Wang and Wu and Xiong and Deyi and Zhang and Enmao Diao and Jie Ding and Vahid Tarokh and Alec Go and Richa Bhayani and Lei Huang and Qiuyi Han and Jie Ding and M Edoardo and Vahid Airoldi and Tarokh and Tian Li and Shengyuan Hu and Ahmad Beirami and Virginia Smith and Wei Yang and Bryan Lim and Cong Nguyen and Dinh Thai Luong and Yutao Hoang and Ying-Chang Jiao and Qiang Liang and Dusit Yang and Chunyan Niyato and Miao and Brendan Mcmahan and Eider Moore and Daniel Ramage and Seth Hampson and Blaise Aguera Y Arcas and Amanda Purington and Jessie Taft and Shruti Sannon and Natalya Bazarova and Samuel Hardman and Taylor and Dan Alistarh and Demjan Grubic and Jerry Li and Ryota Tomioka and Milan Vojnovic and Jeremy Bernstein and Yu-Xiang Wang and Kamyar Azizzadenesheli and Animashree Anandkumar and Theodora Brisimi and Ruidi Chen and Theofanie Mela and Alex Olshevsky and Ioannis Ch Paschalidis and Wei Shi and Mingqing Chen and Ananda Theertha Suresh and Rajiv Mathews and Adeline Wong and Cyril Allauzen and Franoise Beaufays and Michael Riley and Valerii Krzysztof Marcin Choromanski and David Likhosherstov and Xingyou Dohan and Andreea Song and Tamas Gane and Peter Sarlos and Jared Hawkins and Afroz Davis and Lukasz Mohiuddin and David Kaiser and Lucy Belanger},
  title = {{Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.18653/v1/2022.fl4nlp-1},
}
Figure