PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records

Tianyi Zhang, Shirui Zhang, Ziwei Chen, Yoshua Bengio, Dianbo Liu

TL;DR

PMFL (Partial Meta-Federated Learning) is a new framework designed to handle "heterogeneous" tasks—where different participants in a federated network are working on different diagnostic goals. By combining meta-learning with a "partial parameter sharing" strategy, PMFL achieves faster convergence and higher accuracy on real-world medical records compared to traditional federated averaging.

What Problem Does This Paper Address?

Standard Federated Learning (FL), such as the widely used FedAvg algorithm, typically assumes that data is "homogeneous"—meaning every client (like a hospital or a smartphone) is trying to solve the same task using similar data distributions.

In the real world, especially in healthcare, this is rarely true. One hospital might have a dataset focused on pneumonia, while another focuses on lung lesions. This "task heterogeneity" causes standard FL to perform poorly because the global model tries to find a single average set of weights that doesn't actually fit any specific task well. While Meta-Learning (learning "how to learn") is designed for diverse tasks, it is traditionally a local process. The authors tackle the challenge of how to integrate meta-learning into a privacy-preserving federated environment to handle these diverse, non-IID (Independent and Identically Distributed) medical datasets.

Methods

The authors introduce two iterations of their approach:

  1. MetaFL (Meta-Federated Learning): This integrates the Model-Agnostic Meta-Learning (MAML) framework into FL. Instead of clients sending model gradients or weights to the server, they send the training loss from their last local epoch. The server then aggregates these losses to update the global model, effectively training the global model to be a "great starter" that can quickly adapt to any client's specific task.
  2. PMFL (Partial Meta-Federated Learning): To further improve generalization and prevent "catastrophic forgetting," the authors introduce partial parameter sharing. Inspired by sparsification techniques in continual learning, the server only updates the first half of the global model parameters ($\theta[:N/2]$) while keeping the second half frozen.

This "half-frozen" approach allows the model to balance global feature extraction (the shared part) with local task adaptation, ensuring the global model doesn't drift too far toward any single client's distribution.

Datasets

The researchers evaluated PMFL on two major real-world medical datasets:

  • MIMIC-CXR v2.0.0: A massive dataset of chest X-ray radiology reports. They extracted 8 distinct disease labels (e.g., Pneumonia, Atelectasis, Lung Lesion) to simulate 8 heterogeneous tasks across different clients.
  • eICU Collaborative Research Database: Contains granular critical care data from over 200 hospitals. They used drug names and diagnosis data to create text-based patient representations for 7 different disease labels.

Key Insights and Ideas

  • Loss as a Signal: By communicating losses rather than gradients, the server optimizes for the adaptability of the model. It isn't just looking for the average weight; it's looking for the weight initialization that results in the lowest loss across all diverse tasks after a few local steps.
  • The Power of Freezing: The most surprising finding is that updating only half the model (PMFL) significantly outperforms updating the whole model (MetaFL). This acts as a powerful regularizer, preserving "common features" in the frozen weights while allowing the active weights to specialize.
  • Negative Transfer in Standard FL: The results highlight that in highly heterogeneous settings (like eICU), standard Federated Learning can actually be worse than not using FL at all. PMFL fixes this "malignant" interference between tasks.

Key Results

  • Superior Accuracy: PMFL consistently achieved the highest ROC AUC across all test tasks. In the MIMIC-CXR "Pleural Other" task, PMFL reached an AUC of 0.9799, compared to just 0.7166 for standard FL and 0.8702 for a model trained without any federated pre-training.
  • Stability and Speed: PMFL converged much faster than baselines—often within just 4 epochs—and showed significantly lower variance (thinner standard deviation bands in plots) across different experiments.
  • Robustness to Diversity: On the eICU dataset, where tasks were even more divergent, PMFL improved the AUC from a baseline of 0.9218 to 0.9562, while standard FL actually degraded performance to 0.8774.

Limitations and Future Work

  • Task Distance: The authors acknowledge that if the "distance" between tasks is too extreme, even PMFL might struggle. They suggest future research into quantifying the distance between different medical tasks to predict when FL will be effective.
  • Data Imbalance: The current study used silos of relatively similar sizes. In the real world, one hospital might have 10x more data than another. Future work is needed to see how PMFL handles extreme imbalances in client data volume.
  • Communication Overhead: While the paper focuses on performance, meta-learning approaches can sometimes increase the computational burden on local clients; optimizing this for resource-constrained edge devices remains an open area.

Summary updated

Abstract

Federated machine learning is a versatile and flexible tool to utilize distributed data from different sources, especially when communication technology develops rapidly and an unprecedented amount of data could be collected on mobile devices nowadays. Federated learning method exploits not only the data but the computational power of all devices in the network to achieve more efficient model training. Nevertheless, while most traditional federated learning methods work well for homogeneous data and tasks, adapting the method to heterogeneous data and task distribution is challenging. This limitation has constrained the applications of federated learning in real-world contexts, especially in healthcare settings. Inspired by the fundamental idea of meta-learning, in this study we propose a new algorithm, which is an integration of federated learning and meta-learning, to tackle this issue. In addition, owing to the advantage of transfer learning for model generalization, we further improve our algorithm by introducing partial parameter sharing to balance global and local learning. We name this method partial meta-federated learning (PMFL). Finally, we apply the algorithms to two medical datasets. We show that our algorithm could obtain the fastest training speed and achieve the best performance when dealing with heterogeneous medical datasets.
BibTeX
@inproceedings{zhang2022pmfl,
  author = {Tianyi Zhang and Shirui Zhang and Ziwei Chen and Yoshua Bengio and Dianbo Liu},
  title = {{PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=hqmXg6GNHd},
}

Loading related content...

Figure