TransFed: A way to epitomize Focal Modulation using Transformer-based Federated Learning

Tajamul Ashraf, Fuzayil Bin Afzal Mir, Iqra Altaf GillaniWACV 2024 · DOI

Introduction

Federated Learning (FL) has emerged as a vital paradigm for training machine learning models on sensitive medical data, such as chest X-rays, without requiring hospitals to share raw patient information. However, two significant challenges persist: data heterogeneity (non-IID data) and the inherent sensitivity of advanced architectures like Transformers. While Focal Modulation—a powerful alternative to self-attention—excels at capturing long-range dependencies in visual data, its performance often degrades in decentralized settings. Standard aggregation methods like FedAvg tend to disrupt the focal maps that these models rely on to identify critical pathological features.

The TransFed Framework

To address these challenges, TransFed introduces a novel "Learn-to-Tailor" mechanism. Instead of requiring each client to maintain and update full local models, TransFed utilizes a server-side Learnable Generator (a hypernetwork). This small Multi-Layer Perceptron (MLP) takes a low-dimensional "client embedding" as input and generates personalized projection matrices (Query, Key, and Value) for each client's Focal Modulation layers.

The TransFed architecture featuring the Learnable Generator (hypernetwork) that produces personalized projection matrices for focal modulation layers across different clients.

This approach ensures that while core components like MLP blocks and Normalization layers are aggregated globally to foster collaborative learning, the critical feature-extraction layers remain tailored to the specific data distribution of each hospital. This architecture is highly parameter-efficient, as the server only needs to store the hypernetwork and compact client embeddings rather than thousands of full local models.

Visualizing Feature Preservation

A key discovery of this research is that standard federated aggregation (FedAvg) acts as a "feature disruptor." By forcing a single set of parameters across heterogeneous clients, the model's ability to focus on specific medical anomalies is "washed out."

Comparison of focal maps across different clients. Standard FedAvg (FedAvg T) shows disrupted, less localized features compared to tailored approaches, which preserve the model's focus on relevant pathology.

As shown in the visual analysis, TransFed preserves these focal maps, allowing the model to maintain high-fidelity attention on relevant regions in chest X-rays across different client sites. This personalization is crucial for maintaining diagnostic accuracy in the face of varying imaging equipment and patient demographics.

Experimental Results and Insights

The authors evaluated TransFed on the RSNA Pneumonia and Kermany datasets using complex non-IID partitioning strategies. The results demonstrate significant performance gains:

  • Superior Accuracy: On the RSNA dataset with 100 clients, TransFed achieved 92.67% accuracy, vastly outperforming standard FedAvg-T (50.42%) and even surpassing local tailoring methods (91.83%).
  • Rapid Generalization: When introduced to novel clients, TransFed reached 82.6% accuracy within just 3 epochs of fine-tuning, significantly faster than competing methods.
  • Latent Relationships: t-SNE visualizations of the learned client embeddings revealed that the system naturally clusters clients with similar data distributions, even though the server never accesses the raw data.

Conclusion and Future Work

TransFed effectively bridges the gap between the high-performance capabilities of Focal Modulation Transformers and the practical constraints of Federated Learning. By personalizing only the most sensitive projection matrices through a central hypernetwork, it achieves state-of-the-art accuracy while remaining scalable. Future research will focus on reducing the computational overhead of transformer-based training and extending the framework to more complex tasks such as medical image segmentation and object detection.

Summary updated

BibTeX
@inproceedings{ashraf2024transfed,
  author = {Tajamul Ashraf and Fuzayil Bin Afzal Mir and Iqra Altaf Gillani},
  title = {{TransFed: A way to epitomize Focal Modulation using Transformer-based Federated Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00061},
}
Figure