Federated pretraining and fine tuning of BERT using clinical notes from multiple silos

Dianbo Liu, Timothy A. MillerarXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of training powerful language models like BERT on sensitive clinical data, which is often siloed across different institutions due to privacy concerns. They demonstrate that it's possible to both pre-train and fine-tune BERT models using federated learning, where data stays local, achieving competitive results without compromising patient privacy.

What Problem Does This Paper Address?

The advent of large language models (LLMs) like BERT has revolutionized Natural Language Processing (NLP), achieving state-of-the-art results across various tasks. These models learn by being pre-trained on massive text datasets. However, in specialized domains like healthcare, accessing sufficient data for pre-training is a significant hurdle. Clinical data is highly sensitive, protected by strict privacy regulations (like HIPAA) and institutional policies, making it extremely difficult to aggregate from multiple hospitals or healthcare systems. This "data siloing" prevents the creation of robust, domain-specific models that could significantly benefit medical research and practice. The paper aims to bridge this gap by enabling federated learning for BERT, allowing models to learn from distributed clinical data without it ever leaving its original location.

Methods

The core of this paper is the application of federated learning to the BERT model. Federated learning is a distributed machine learning approach where the model is trained across multiple decentralized edge devices or servers holding local data samples, without exchanging their data samples.

Here's how they applied it:

  • Federated Pre-training: They took the publicly available MIMIC-III corpus and simulated "silos" by splitting the data. BERT models were initialized with identical parameters and sent to each silo. Each silo then trained the model on its local data for one epoch. Only the updated model parameters were sent back to a central server for aggregation. This process was repeated for multiple global cycles. The aggregation involved averaging the parameters from each silo, weighted by the number of patients in that silo.
  • Federated Fine-tuning: After federated pre-training, the resulting model was then fine-tuned for a specific downstream task (Named Entity Recognition - NER) using a similar federated approach. Here, the data was split by notes, and the aggregation was weighted by the number of notes in each silo.

They adapted the pre-processing and tokenization pipeline from previous work and used a standard BERT architecture. The key innovation is the federated application of both pre-training and fine-tuning stages.

Datasets

  • MIMIC-III Corpus: This publicly available dataset was used for both federated pre-training and, in some experiments, for centralized pre-training. It contains de-identified health information for over 58,000 admissions from more than 45,000 patients at Beth Israel Deaconess Medical Center between 2001 and 2012. The authors specifically focused on using discharge summaries for pre-training, as previous studies showed this yielded comparable performance to using all note types.
  • i2b2 2010 and i2b2 2012 Datasets: These datasets were used for evaluating the performance of the fine-tuned models on the Named Entity Recognition (NER) task. These are standard benchmarks for clinical NER.

Key Insights and Ideas

  • Feasibility of Federated Clinical BERT: The most significant insight is that it's demonstrably possible to pre-train and fine-tune BERT models on clinical text in a federated manner, without data ever leaving its silo. This opens the door to leveraging vast amounts of sensitive clinical data that were previously inaccessible for model training.
  • "Federated Communication Loss": The authors identify a performance drop when data is distributed and models are trained in a federated way, which they term "federated communication loss." This loss is attributed to the challenges of distributed training and communication overhead.
  • Impact of Federated Stages: They found that federated pre-training followed by centralized fine-tuning resulted in a smaller performance drop (less than 5%) compared to fully centralized training. Conversely, centralized pre-training followed by federated fine-tuning had an even smaller drop (less than 2%). However, when both pre-training and fine-tuning were conducted in a federated manner, the performance drop was more noticeable (around 6%). This suggests that the pre-training stage is more sensitive to the federated approach.
  • Attention Head Behavior: Analysis of attention heads revealed that the federated clinical BERT model's attention patterns were more similar to the original BERTbase model than to a centrally trained clinical BERT model. This suggests that federated training might preserve certain general linguistic characteristics better than centralized training on a specific domain, or that the centralized training might be "overfitting" to the specific silo's data distribution.

Key Results

The paper presents performance metrics (F1 scores) for various experimental setups on the i2b2 NER tasks. Here are some highlights:

  • Centralized Clinical BERT vs. Original BERT: Centralized pre-training and fine-tuning on MIMIC-III data (ClinicalBERT) significantly improved performance over the original BERTbase model. For i2b2 2010, F1 scores were 0.858 (ClinicalBERT) vs. 0.784 (BERTbase).
  • Federated Pre-training + Centralized Fine-tuning: This setup achieved an F1 score of 0.820 for i2b2 2010, showing a good performance despite data being distributed during pre-training.
  • Federated Pre-training + Federated Fine-tuning: This fully federated approach yielded F1 scores of 0.808 for i2b2 2010 and 0.715 for i2b2 2012. While slightly lower than fully centralized methods, these results are still superior to the original BERT model without domain-specific pre-training.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simulated Silos: The study used data from a single healthcare system (MIMIC-III) and simulated different silos. Future work should involve actual data from diverse healthcare providers to better reflect real-world scenarios.
  • Data Overlap: There's a potential overlap between the MIMIC-III training data and the i2b2 NER tasks, which could inflate performance metrics. They avoided using MedNLI tasks for this reason. Future studies should use datasets with minimal overlap.

Promising future directions include:

  • Exploring more sophisticated federated learning algorithms to mitigate the "federated communication loss."
  • Investigating the impact of different silo configurations and data distributions on model performance.
  • Applying this federated approach to a wider range of clinical NLP tasks and datasets.

Summary updated

Abstract

Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
BibTeX
@misc{liu2020federated,
  author = {Dianbo Liu and Timothy A. Miller},
  title = {{Federated pretraining and fine tuning of BERT using clinical notes from multiple silos}},
  year = {2020},
  eprint = {2002.08562},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.08562},
}
Figure