Federated learning of molecular properties with graph neural networks in a heterogeneous setting

Wei Zhu, Jiebo Luo, Andrew D. WhitePatterns 2021 · DOI · arXiv · OpenReview

TL;DR

This paper introduces FedChem, a new benchmark for federated learning on molecular data, which is inherently heterogeneous across different research institutions. To address the performance degradation caused by this heterogeneity, they propose FLIT(+) (Federated Learning by Instance Reweighting), a novel method that reweights training samples to better align local models with the global objective, leading to improved performance.

What Problem Does This Paper Address?

Training machine learning models for molecular property prediction is crucial for accelerating drug discovery and materials science, but it's often hampered by the high cost of experiments and computation. While graph neural networks (GNNs) have shown great promise, a major hurdle is data sharing. Different institutions often possess proprietary datasets focused on specific types of molecules, making it difficult to pool data for centralized training. Federated learning offers a solution by allowing collaborative model training without sharing raw data. However, existing federated learning methods struggle when the data is heterogeneously distributed across clients (i.e., each client has data from a different subset of molecules), leading to significant performance drops. This paper tackles this challenge by proposing a realistic benchmark for this scenario and a method to overcome it.

Methods

The authors introduce FedChem, a benchmark for federated heterogeneous molecular learning. They simulate heterogeneity by:

  1. Scaffold Splitting: Grouping molecules based on their 2D structural similarity.
  2. Latent Dirichlet Allocation (LDA): Assigning these scaffold groups to different clients in an unbalanced manner, mimicking how institutions might specialize in certain molecular families.

To combat the performance degradation caused by this heterogeneity, they propose Federated Learning by Instance Reweighting (FLIT) and its extension FLIT(+):

  • FLIT: This method adapts the concept of focal loss for federated learning. The core idea is that local models, trained on their own biased data, tend to overfit common samples and become overconfident, while being uncertain about rarer ones. FLIT reweights training samples, giving more importance to uncertain samples by considering both the local and global models. This helps align the local training distributions with the global one.
  • FLIT(+): This builds upon FLIT by using a more sophisticated measure of uncertainty. Instead of just using the loss value, FLIT(+) also incorporates the discrepancy between a sample's prediction and its "adversarial neighbors" (samples with similar input but very different predictions). This aims to improve neighborhood prediction consistency.

Datasets

The experiments were conducted on nine datasets from MoleculeNet [57], including:

  • Regression datasets: FreeSolv, Lipophilicity, ESOL, and QM9.
  • Classification datasets: Tox21, SIDER, ClinTox, BBBP, and BACE.

The paper also introduces FedChem as a simulated benchmark for heterogeneous federated molecular learning, generated from these existing datasets using scaffold splitting and LDA.

Key Insights and Ideas

  • Heterogeneity is a major challenge: Standard federated learning methods like FedAvg perform poorly on molecular data when it's distributed heterogeneously across clients, highlighting the need for specialized approaches.
  • Instance reweighting is key: FLIT(+)'s success stems from its ability to identify and focus on uncertain samples, effectively bridging the gap between local, biased data distributions and the desired global model.
  • Leveraging the global model is crucial: FLIT(+)'s effectiveness is enhanced by incorporating the global model's knowledge into the local training objective, not just for aggregation but also for guiding sample weighting.
  • Adversarial neighbors add value: FLIT(+)'s use of adversarial neighbors to measure prediction discrepancy provides a more robust uncertainty estimate, leading to better performance.

Key Results

  • FedChem benchmark: The proposed FedChem benchmark demonstrates significant performance degradation for standard federated learning methods on 7 out of 9 datasets.
  • FLIT(+) outperforms baselines: FLIT(+) consistently achieves better performance than FedAvg, FedProx, and MOON across various datasets. For instance, on FreeSolv, FLIT(+) achieved improvements of 0.543 and 0.162 for different heterogeneity settings.
  • FLIT(+) is robust: The method shows consistent improvements across different heterogeneity levels (controlled by the LDA parameter 'a').
  • FLIT(+) is efficient: The paper notes that the additional computations for FLIT(+) are manageable, as they primarily involve calculations with the global model, which is done once per communication round.

Limitations and Future Work

  • Scaffold splitting limitations: The authors acknowledge that their current scaffold splitting method might not always create sufficiently severe structural heterogeneity in all cases. They plan to explore other approaches for simulating more challenging heterogeneous settings.
  • Personalized Federated Learning: The paper suggests that developing personalized federated learning for FedChem is a promising future direction.
  • Generalizability: While FLIT(+) shows robustness, the authors mention that further work is needed to explore other approaches for more heterogeneous settings.

Summary updated

Abstract

Chemistry research has both high material and computational costs to conduct experiments. Institutions thus consider chemical data to be valuable and there have been few efforts to construct large public datasets for machine learning. Another challenge is that different intuitions are interested in different classes of molecules, creating heterogeneous data that cannot be easily joined by conventional distributed training. In this work, we introduce federated heterogeneous molecular learning to address these challenges. Federated learning allows end-users to build a global model collaboratively while keeping the training data distributed over isolated clients. Due to the lack of related research, we first simulate a heterogeneous federated learning benchmark (FedChem) by jointly performing scaffold splitting and latent Dirichlet allocation on existing datasets for heterogeneously distributed client data. Our results on FedChem show that significant learning challenges arise when working with heterogeneous molecules across clients. We then propose a method to alleviate the problem, namely Federated Learning by Instance reweighTing (FLIT(+)). FLIT(+) can align the local training across heterogeneous clients by improving the performance for uncertain samples. Comprehensive experiments conducted on our new benchmark FedChem validate the advantages of this method over other federated learning schemes. FedChem should enable a new type of collaboration for improving AI in chemistry that mitigates concerns about valuable chemical data.
BibTeX
@inproceedings{zhu2021federated,
  author = {Wei Zhu and Jiebo Luo and Andrew D. White},
  title = {{Federated learning of molecular properties with graph neural networks in a heterogeneous setting}},
  booktitle = {Patterns},
  year = {2021},
  doi = {10.48550/arxiv.2109.07258},
}

Loading related content...

Figure