Abstract
Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy. However, the non-identically and independently distributed (non-IID) nature of real-world data often necessitates Personalized Federated Learning (PFL). Current PFL methods frequently suffer from a "catastrophic forgetting" of global knowledge during local personalization. We introduce Personalized Semantics Excitation (PSE), a framework that utilizes Grad-CAM to identify critical features and a cross-model attention mechanism to fuse global semantics into local models. PSE achieves state-of-the-art accuracy and superior fairness across diverse benchmarks.
Introduction
The core challenge in Federated Learning is data heterogeneity. A global model trained on diverse datasets (e.g., different dog breeds across different climates) often fails to perform optimally for any single client. Personalized Federated Learning (PFL) addresses this by allowing clients to adapt the global model to their local distributions.
However, this adaptation often leads to a "tug-of-war" between local and global knowledge. When a client personalizes too aggressively, they may override essential discriminative features learned by the global model—such as the general shape of an animal's face—replacing them with noise or overfitted local patterns. Existing methods like Ditto or FedRep often struggle to maintain this balance, leading to degraded performance on specific features.
Methodology: Personalized Semantics Excitation (PSE)
PSE is designed to "excite" and preserve global knowledge within the local training process through two primary modules:
1. Adaptive Personalized Channel Excitation
Instead of uniform regularization, PSE employs Grad-CAM to surgically identify which convolutional channels are most discriminative for a given image. By comparing importance scores between the global and local models, PSE selectively injects knowledge from the global model's channels into the local model only when the global signal is stronger. This prevents the local model from losing focus on critical semantic parts (e.g., a dog's nose or mouth) that are essential for classification.
2. Cross-Model Attention Enhancement
In the network's final layers, PSE implements a cross-model attention module. Unlike standard self-attention, this module exchanges "Queries" between local and global feature maps. This allows the local classifier to "borrow" high-level semantic representations from the global model, enhancing robustness. To maintain efficiency for edge devices, the authors utilize lightweight 1D convolutional filters for attention projections.
Experimental Results
Performance and Robustness
PSE was evaluated on CIFAR-10/100, FEMNIST, and FashionMNIST, including scenarios with significant visual discrepancies (e.g., color vs. edge-detected images). On CIFAR-100 with 50 clients, PSE achieved 62.46% accuracy, significantly outperforming FedRep (58.94%) and Ditto (56.14%).
Fairness and Convergence
A critical metric in FL is fairness—the consistency of performance across all participating clients. PSE reduces the performance gap between "easy" and "hard" clients, resulting in a lower standard deviation of accuracy across the network. Furthermore, empirical evidence confirms that PSE converges stably despite its architectural enhancements.
Conclusion
Personalized Semantics Excitation (PSE) provides a surgical approach to PFL, ensuring that local adaptation does not come at the cost of fundamental global knowledge. By leveraging gradient-based importance and cross-model attention, PSE sets a new standard for accuracy and fairness in federated image classification. Future work will explore the scalability of this semantic excitation approach to Transformer-based architectures and Large Language Models.
Summary updated