PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records

Tianyi Zhang, Shirui Zhang, Ziwei Chen, Dianbo Liu

Abstract

Federated machine learning is a versatile tool to utilize distributed data, but traditional methods struggle with heterogeneous data and task distributions, which are common in healthcare. We propose Partial Meta-Federated Learning (PMFL), integrating meta-learning with partial parameter sharing to address this. Evaluated on real-world medical datasets, PMFL achieves superior performance and faster training speeds compared to standard federated learning.

Introduction

Standard Federated Learning (FL), such as the widely used FedAvg algorithm, typically assumes that data is "homogeneous"—meaning every client (e.g., a hospital) is trying to solve the same task using similar data distributions. In the real world, especially in healthcare, this is rarely true. One hospital might have a dataset focused on pneumonia, while another focuses on lung lesions. This "task heterogeneity" causes standard FL to perform poorly because the global model tries to find a single average set of weights that doesn't actually fit any specific task well.

Methodology

The authors propose two advancements to bridge the gap between federated learning and diverse real-world tasks:

Meta-Federated Learning (MetaFL)

MetaFL integrates the Model-Agnostic Meta-Learning (MAML) framework into FL. Instead of clients sending model gradients or weights to the server, they communicate the training loss from their local epochs. The server aggregates these losses to optimize the global model's adaptability, effectively training it to be a "great starter" that can quickly adapt to any client's specific task.

Partial Meta-Federated Learning (PMFL)

To further improve generalization and prevent "catastrophic forgetting," PMFL introduces partial parameter sharing. Inspired by sparsification techniques in continual learning, the server only updates a subset of the global model parameters (e.g., half the nodes) while keeping the rest frozen. This "half-frozen" approach allows the model to balance global feature extraction with local task adaptation, ensuring the global model doesn't drift too far toward any single client's distribution.

The PMFL workflow: Server distributes the global model, clients return training losses, and the server updates the model while freezing half of the nodes.

Experimental Evaluation

The researchers evaluated PMFL on two major real-world medical datasets:

  • MIMIC-CXR v2.0.0: A massive dataset of chest X-ray radiology reports with 8 distinct disease labels (e.g., Pneumonia, Atelectasis) simulating heterogeneous tasks.
  • eICU Collaborative Research Database: Granular critical care data from over 200 hospitals, used to create patient representations for 7 different disease labels.

Results and Discussion

PMFL consistently outperformed all baselines in accuracy, stability, and speed across both datasets.

  • Superior Accuracy: In the MIMIC-CXR "Pleural Other" task, PMFL reached an AUC of 0.9799, compared to just 0.7166 for standard FL and 0.8702 for a model trained without any federated pre-training.
  • Convergence Speed: PMFL converged much faster than baselines—often within just 4 epochs—and showed significantly lower variance across different experiments.
  • Mitigating Negative Transfer: On the eICU dataset, where tasks were even more divergent, standard FL actually performed worse than no FL at all (0.8774 vs 0.9218). PMFL corrected this "malignant" interference, reaching an AUC of 0.9562.
Test AUC performance on the Pneumonia task, showing PMFL (red) outperforming MetaFL (green), standard FL (blue), and local training (black).

The success of PMFL highlights the power of using "loss as a signal" for adaptability and the importance of parameter freezing as a regularizer in diverse environments.

Conclusion

PMFL provides a robust solution for federated learning in heterogeneous environments. By combining the adaptability of meta-learning with the stability of partial parameter sharing, it enables effective collaborative training across diverse medical tasks without compromising privacy or performance.

Summary updated

Abstract

Federated machine learning is a versatile and flexible tool to utilize distributed data from different sources, especially when communication technology develops rapidly and an unprecedented amount of data could be collected on mobile devices nowadays. Federated learning method exploits not only the data but the computational power of all devices in the network to achieve more efficient model training. Nevertheless, while most traditional federated learning methods work well for homogeneous data and tasks, adapting the method to a different heterogeneous data and task distribution is challenging. This limitation has constrained the applications of federated learning in real-world contexts, especially in healthcare settings. Inspired by the fundamental idea of meta-learning, in this study we propose a new algorithm, which is an integration of federated learning and meta-learning, to tackle this issue. In addition, owing to the advantage of transfer learning for model generalization, we further improve our algorithm by introducing partial parameter sharing. We name this method partial meta-federated learning (PMFL). Finally, we apply the algorithms to two medical datasets. We show that our algorithm could obtain the fastest training speed and achieve the best performance when dealing with heterogeneous medical datasets.
BibTeX
@inproceedings{zhang2022pmfl,
  author = {Tianyi Zhang and Shirui Zhang and Ziwei Chen and Dianbo Liu},
  title = {{PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=hqmXg6GNHd},
}
Figure