An in-depth evaluation of federated learning on biomedical natural language processing for information extraction

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Le Peng, Gaoxiang Luo, Sicheng Zhou, Jiandong Chen, Ziyue Xu, Ju Sun, Rui Zhang

TL;DR

This paper investigates the effectiveness of Federated Learning (FL) for biomedical Natural Language Processing (NLP) tasks, demonstrating that FL models can match or even surpass centralized training performance while preserving data privacy. The study highlights FL's superiority over large language models (LLMs) with few-shot prompting in this domain and explores the impact of model size and data heterogeneity on FL performance.

What Problem Does This Paper Address?

Training effective Natural Language Processing (NLP) models, particularly large language models (LLMs), typically requires vast amounts of data. However, in sensitive domains like healthcare, accessing and aggregating this data is severely restricted due to privacy regulations (e.g., HIPAA, GDPR). This limitation hinders the development of specialized biomedical NLP models. While federated learning (FL) offers a decentralized approach to train models collaboratively without sharing raw data, its application and effectiveness in the complex and heterogeneous biomedical domain, especially for information extraction tasks, remain underexplored. This paper aims to bridge this gap by providing a comprehensive evaluation of FL for biomedical NLP.

Methods

The study evaluates Federated Learning (FL) using two primary algorithms: FedAvg (Federated Averaging), a standard approach that averages model weights from clients, and FedProx, which adds a proximal term to the local objective function to mitigate issues arising from heterogeneous local updates. These FL algorithms are applied to train various language models (LMs), including transformer-based models like BERT, BlueBERT, BioBERT, Bio_ClinicalBERT, and GPT-2, as well as a classical BiLSTM-CRF model. The evaluation focuses on two key biomedical NLP tasks: Named Entity Recognition (NER) and Relation Extraction (RE). The researchers compare FL performance against two baselines: single-client learning (training a model on data from only one client) and centralized learning (training a model on all data pooled together). Additionally, they compare FL models against large language models (LLMs) like GPT-4, PaLM 2, and Gemini Pro using few-shot prompting.

Datasets

The study utilizes a collection of 8 biomedical corpora for training and evaluation across NER and RE tasks:

  • NER Datasets: 2018 n2c2, BC2GM, BC4CHEMD, JNLPBA, NCBI-disease.
  • RE Datasets: 2018 n2c2, EUADR, GAD.

These datasets are publicly available and represent a diverse range of biomedical text types and annotation schemes. The paper does not introduce new benchmark datasets but rather uses existing ones to provide a comprehensive evaluation.

Key Insights and Ideas

  • FL as a Privacy-Preserving Powerhouse: FL consistently outperforms single-client learning and often achieves performance comparable to centralized learning, proving its efficacy for privacy-preserving model training in the biomedical domain.
  • LLMs Aren't Always the Answer (Yet): For biomedical information extraction, FL models trained on domain-specific data significantly outperform general LLMs using few-shot prompting. This suggests that domain shift is a significant challenge for LLMs, and FL can effectively overcome it.
  • Model Size Matters, But Resilience Varies: Larger, transformer-based models (like BERT and GPT-2) show greater resilience to changes in FL scale (number of clients) and data heterogeneity compared to smaller, classical models (BiLSTM-CRF).
  • Data Heterogeneity is a Real Challenge, But Manageable: While non-IID data can impact performance, FedProx shows promise in mitigating these effects, especially when its hyperparameter is carefully tuned.
  • GPT-2's Weakness in this Domain: GPT-2 consistently underperformed BERT-based models, likely due to its unidirectional attention mechanism, which is less suited for capturing the complex dependencies in biomedical text compared to BERT's bidirectional approach.

Key Results

  • FedAvg vs. Centralized/Single-Client: In most cases, FedAvg achieved performance on par with or better than centralized learning and significantly better than single-client learning across both NER and RE tasks. For instance, on BC4CHEMD and 2018 n2c2 datasets, FedAvg models matched centralized learning performance.
  • Impact of FL Scale: With a fixed amount of data, increasing the number of clients generally degraded FL model performance. However, this degradation was less pronounced for larger, pre-trained transformer models.
  • FedProx vs. FedAvg on Non-IID Data: FedProx demonstrated slightly better performance than FedAvg on non-IID data, though its sensitivity to the hyperparameter $\mu$ was noted.
  • FL vs. LLMs: FL models consistently outperformed LLMs (GPT-4, PaLM 2, Gemini Pro) using few-shot prompting across NER and RE tasks. For example, in Fig. 3, FL models (BlueBERT and GPT-2) showed a clear advantage over LLMs even with 20-shot prompting.
  • Model Architecture Impact: BioBERT emerged as a top performer among the evaluated models, indicating the benefit of domain-specific pre-training. GPT-2 generally yielded inferior results compared to BERT-based models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simulated Environment: Most experiments, except for the non-IID study, were conducted in a simulated environment with synthetic data splits, which may not perfectly reflect real-world FL data distributions.
  • Focus on Horizontal FL: The study primarily focused on horizontal FL and did not explore vertical FL.
  • Lack of Advanced Privacy Techniques: Privacy techniques like differential privacy and homomorphic encryption were not integrated into the FL framework.

Future work suggested includes:

  • Real-world implementation of FL for biomedical NLP.
  • Exploring practical opportunities and challenges in FL, such as vertical FL and integrating advanced privacy techniques.
  • Developing more effective AI systems by leveraging distributed clinical data.

Summary updated

Abstract

Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: (1) FL models consistently outperformed models trained on individual clients’ data and sometimes performed comparably with models trained with polled data; (2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. (3) FL models significantly outperformed pre-trained LLMs with few-shot prompting.
BibTeX
@article{fiahsi2024depth,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Le Peng and Gaoxiang Luo and Sicheng Zhou and Jiandong Chen and Ziyue Xu and Ju Sun and Rui Zhang},
  title = {{An in-depth evaluation of federated learning on biomedical natural language processing for information extraction}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01126-4},
}
Figure