Personalized Semantics Excitation for Federated Image Classification

Haifeng Xia, Kai Li, Zhengming Ding

Abstract

Federated Learning (FL) enables collaborative model training across decentralized clients while preserving data privacy. However, the non-identically and independently distributed (non-IID) nature of real-world data often necessitates Personalized Federated Learning (PFL). Current PFL methods frequently suffer from a "catastrophic forgetting" of global knowledge during local personalization. We introduce Personalized Semantics Excitation (PSE), a framework that utilizes Grad-CAM to identify critical features and a cross-model attention mechanism to fuse global semantics into local models. PSE achieves state-of-the-art accuracy and superior fairness across diverse benchmarks.

Introduction

The core challenge in Federated Learning is data heterogeneity. A global model trained on diverse datasets (e.g., different dog breeds across different climates) often fails to perform optimally for any single client. Personalized Federated Learning (PFL) addresses this by allowing clients to adapt the global model to their local distributions.

However, this adaptation often leads to a "tug-of-war" between local and global knowledge. When a client personalizes too aggressively, they may override essential discriminative features learned by the global model—such as the general shape of an animal's face—replacing them with noise or overfitted local patterns. Existing methods like Ditto or FedRep often struggle to maintain this balance, leading to degraded performance on specific features.

Methodology: Personalized Semantics Excitation (PSE)

PSE is designed to "excite" and preserve global knowledge within the local training process through two primary modules:

1. Adaptive Personalized Channel Excitation

Instead of uniform regularization, PSE employs Grad-CAM to surgically identify which convolutional channels are most discriminative for a given image. By comparing importance scores between the global and local models, PSE selectively injects knowledge from the global model's channels into the local model only when the global signal is stronger. This prevents the local model from losing focus on critical semantic parts (e.g., a dog's nose or mouth) that are essential for classification.

Grad-CAM visualizations comparing PSE (Ours) with other PFL methods. PSE maintains focus on critical discriminative features like the face of the dog, which other methods often lose during personalization.

2. Cross-Model Attention Enhancement

In the network's final layers, PSE implements a cross-model attention module. Unlike standard self-attention, this module exchanges "Queries" between local and global feature maps. This allows the local classifier to "borrow" high-level semantic representations from the global model, enhancing robustness. To maintain efficiency for edge devices, the authors utilize lightweight 1D convolutional filters for attention projections.

Experimental Results

Performance and Robustness

PSE was evaluated on CIFAR-10/100, FEMNIST, and FashionMNIST, including scenarios with significant visual discrepancies (e.g., color vs. edge-detected images). On CIFAR-100 with 50 clients, PSE achieved 62.46% accuracy, significantly outperforming FedRep (58.94%) and Ditto (56.14%).

Fairness and Convergence

A critical metric in FL is fairness—the consistency of performance across all participating clients. PSE reduces the performance gap between "easy" and "hard" clients, resulting in a lower standard deviation of accuracy across the network. Furthermore, empirical evidence confirms that PSE converges stably despite its architectural enhancements.

(a) Fairness analysis showing PSE's consistent performance across diverse clients compared to FedRep and Local models. (b) Convergence analysis demonstrating stable training loss across different CIFAR configurations.

Conclusion

Personalized Semantics Excitation (PSE) provides a surgical approach to PFL, ensuring that local adaptation does not come at the cost of fundamental global knowledge. By leveraging gradient-based importance and cross-model attention, PSE sets a new standard for accuracy and fairness in federated image classification. Future work will explore the scalability of this semantic excitation approach to Transformer-based architectures and Large Language Models.

Summary updated

Abstract

Federated learning casts a light on the collaboration of distributed local clients with privacy protected to attain a more generic global model. However, significant distribution shift in input/label space across different clients makes it challenging to well generalize to all clients, which motivates personalized federated learning (PFL). Existing PFL methods typically customize the local model by fine-tuning with limited local supervision and the global model regularizer, which secures local specificity but risks ruining the global discriminative knowledge. In this paper, we propose a novel Personalized Semantics Excitation (PSE) mechanism to breakthrough this limitation by exciting and fusing personalized semantics from the global model during local model customization. Specifically, PSE explores channel-wise gradient differentiation across global and local models to identify important low-level semantics mostly from convolutional layers which are embedded into the client-specific training. In addition, PSE deploys the collaboration of global and local models to enrich high-level feature representations and facilitate the robustness of client classifier through a cross-model attention module. Extensive experiments and analysis on various image classification benchmarks demonstrate the effectiveness and advantage of our method over the state-of-the-art PFL methods.
BibTeX
@inproceedings{xia2023personalized,
  author = {Haifeng Xia and Kai Li and Zhengming Ding},
  title = {{Personalized Semantics Excitation for Federated Image Classification}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01768},
}
Figure