Federated Learning for Chronic Obstructive Pulmonary Disease Classification with Partial Personalized Attention Mechanism

Yiqing Shen, Baiyun Liu, Ruize Yu, Yudong Wang, Shaokang Wang, Jiangfen Wu, Weidao Chen

TL;DR

This paper introduces a novel federated learning approach for diagnosing Chronic Obstructive Pulmonary Disease (COPD) from CT scans using Vision Transformers (ViTs). By partially personalizing the attention mechanisms within ViTs, the method effectively addresses data heterogeneity across different medical centers while preserving patient privacy, outperforming existing federated learning techniques.

What Problem Does This Paper Address?

Diagnosing COPD, a leading cause of death, is challenging due to its heterogeneous nature and the limitations of traditional spirometry, often leading to underdiagnosis, especially in early stages. Deep learning models show promise for COPD detection from CT scans, but training these models centrally is hindered by privacy regulations and the "Non-IID" (non-independent and identically distributed) nature of data from different hospitals. Federated learning (FL) offers a solution by training models locally without sharing raw data, but standard FL struggles with data heterogeneity, leading to performance degradation. This paper tackles the problem of effectively training personalized FL models for COPD detection using ViTs, which are known for their ability to handle complex image data, while accounting for the inherent data variations across different medical institutions.

Methods

The core innovation lies in a partial personalization scheme for the Multi-Head Self-Attention (MSA) layers within Vision Transformers (ViTs). Instead of personalizing the entire model or specific layers, this approach selectively personalizes a subset of attention heads in each MSA layer. These personalized heads learn local, client-specific attention patterns tailored to individual data distributions. The remaining attention heads are kept globally shared, allowing the model to learn common, generalizable attention features across all clients. This hybrid approach aims to balance local adaptation with global knowledge sharing.

To further enhance this, the authors introduce a consistency regularization term. This regularization encourages the locally personalized ViT subnets and the globally shared ViT subnet to produce similar predictions, ensuring that personalization doesn't lead to a complete divergence from the common learned features. The overall objective function combines the standard cross-entropy loss for classification with this consistency regularization term, balanced by a hyperparameter.

The framework uses a standard ViT backbone for processing CT image patches and employs the FEDAVG algorithm for aggregating the globally shared model weights.

Datasets

The study utilizes a dataset curated from six medical centers, including the public NLST dataset (Center 1) and data from five additional hospitals (Centers 2-6). The primary inclusion criterion for COPD diagnosis was a forced expiratory volume in 1 sec (FEV1)/forced vital capacity (FVC) ratio of < 0.7. The dataset comprises a total of 2856 participants, with 1077 diagnosed with COPD and 1779 with non-COPD. The paper provides details on the data scale and distributions across these centers, highlighting the heterogeneity in patient numbers and class distribution.

Key Insights and Ideas

  • ViTs are well-suited for federated learning in medical imaging: The paper demonstrates that ViTs, with their attention mechanisms, can effectively capture complex patterns in medical images, making them a strong candidate for federated learning scenarios where data is distributed and heterogeneous.
  • Partial personalization is key for Non-IID data: The insight that personalizing only a portion of the attention heads, rather than the entire model or specific blocks, strikes a crucial balance. This allows for local adaptation without sacrificing the benefits of global knowledge sharing, which is critical for Non-IID data.
  • Attention heads as a flexible personalization target: Targeting specific attention heads within the MSA layers offers a fine-grained and effective way to achieve personalization. Different heads can learn different aspects of the data, allowing for both specialized and generalized feature extraction.
  • Consistency regularization bridges local and global learning: The regularization term is vital for ensuring that the personalized models remain grounded in the common knowledge learned by the global model, preventing overfitting to local data and improving overall generalization.

Key Results

The proposed method demonstrates significant improvements over baseline federated learning (FEDAVG) and local training approaches. Specifically:

  • The method improves baseline FEDAVG by 0.012 to 0.065 in local test AUC and 0.049 in new test AUC.
  • It outperforms state-of-the-art CNN-based personalized federated learning methods (FEDPER and LG-FEDAVG) in terms of local test AUC.
  • The approach achieves the highest new test AUC, indicating its strong capability in extracting common attention features while also benefiting from local personalization.
  • Ablation studies show that the optimal personalization ratio (p) for local test AUC is around 0.6, and for new test AUC, it's around 0.4, suggesting that a competitive proportion of personalized knowledge is beneficial.
  • The consistent regularization term (Lcon) further enhances performance, confirming its effectiveness.

Limitations and Future Work

  • Hyperparameter sensitivity: The paper notes that the performance is influenced by the personalization ratio (p) and suggests that finding the optimal ratio is important. The consistent regularization coefficient (λ) also plays a role.
  • Future Work: The authors suggest evaluating their approach on different data modalities as a promising future direction. This implies exploring the applicability of their partial personalization scheme beyond CT scans to other types of medical imaging data.

Summary updated

Abstract

Chronic Obstructive Pulmonary Disease (COPD) is the fourth leading cause of death worldwide. Yet, COPD diagnosis heavily relies on spirometric examination as well as functional airway limitation, which may cause a considerable portion of COPD patients underdiagnosed especially at the early stage. Recent advance in deep learning (DL) has shown their promising potential in COPD identification from CT images. However, with heterogeneous syndromes and distinct phenotypes, DL models trained with CTs from one data center fail to generalize on images from another center. Due to privacy regularizations, a collaboration of distributed CT images into one centralized center is not feasible. Federated learning (FL) approaches enable us to train with distributed private data. Yet, routine FL solutions suffer from performance degradation in the case where COPD CTs are not independent and identically distributed (Non-IID). To address this issue, we propose a novel personalized federated learning (PFL) method based on vision transformer (ViT) for distributed and heterogeneous COPD CTs. To be more specific, we partially personalize some heads in multiheaded self-attention layers to learn the personalized attention for local data and retain the other heads shared to extract the common attention. To the best of our knowledge, this is the first proposal of a PFL framework specifically for ViT to identify COPD. Our evaluation of a dataset set curated from six medical centers shows our method outperforms the PFL approaches for convolutional neural networks.
BibTeX
@inproceedings{shen2022federated,
  author = {Yiqing Shen and Baiyun Liu and Ruize Yu and Yudong Wang and Shaokang Wang and Jiangfen Wu and Weidao Chen},
  title = {{Federated Learning for Chronic Obstructive Pulmonary Disease Classification with Partial Personalized Attention Mechanism}},
  booktitle = {BIBM},
  year = {2022},
  url = {https://openreview.net/forum?id=oRq5Eo7ht3},
}
Figure