Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data

Zhuang Qi, Lei Meng, Zitan Chen, Han Hu, Hui Lin, Xiangxu Meng

Introduction

Federated Learning (FL) has emerged as a vital paradigm for training machine learning models across decentralized data sources while maintaining privacy. However, a significant hurdle remains: the problem of non-IID (not Independent and Identically Distributed) data. In real-world scenarios, such as medical imaging across different hospitals, data distributions vary wildly in terms of volume, quality, and characteristics. This heterogeneity leads to "local drift," where individual client models develop inconsistent feature representations.

This paper introduces FedCSPC (Cross-Silo Prototypical Calibration), a framework designed to bridge these feature gaps. By aligning inconsistent feature spaces through clustering-based prototypes and server-side contrastive alignment, FedCSPC creates a unified representation space that significantly enhances model generalization.

The Challenge of Feature Inconsistency

In standard Federated Learning approaches like FedAvg, the central server aggregates local model weights to form a global model. This process assumes that a specific feature (e.g., the visual characteristics of a "cat") is represented similarly across all clients. In reality, non-IID data causes each client to develop its own "internal language" or feature space. A feature vector that represents a specific class in one silo might be mapped to a completely different class in another.

When these mismatched models are averaged, the resulting global model often suffers from poor performance because it cannot reconcile these divergent feature definitions. FedCSPC addresses this by focusing on the "feature gap" rather than just weight bias.

The FedCSPC Methodology

FedCSPC introduces a multi-stage calibration process to ensure that feature representations remain consistent across the entire network.

1. Data Prototypical Modeling (DPM)

Traditional prototype-based methods often represent a class using a single mean vector. However, real-world data is often multi-modal (e.g., different breeds of dogs or different lighting conditions for the same object). FedCSPC utilizes K-means clustering on local client data to identify multiple "prototypes" per class. This captures the diversity within each category, providing a more nuanced map of the local feature space to the server.

2. Cross-Silo Prototypical Calibration (CSPC)

On the server side, these prototypes are used to re-calibrate the global model through Augmented Contrastive Learning. This involves two key techniques:

  • Positive Mixing: The server generates synthetic samples by blending prototypes of the same class, increasing the diversity of the representation.
  • Hard Negative Mining: The server creates challenging "near-miss" samples by interpolating between different classes. This forces the model to learn sharper, more distinct decision boundaries.

3. Triple-Level Regularization

To prevent local models from drifting too far during training, FedCSPC applies three levels of constraints:

  • Node-level: Ensures the distance between local features and global prototypes is minimized.
  • Angle-level: Aligns the orientation of feature vectors.
  • Edge-level: Maintains the relative distances between different class representations.

4. Knowledge-based Prediction (KP)

During inference, the system employs a Knowledge-based Prediction module. This acts as a secondary verification step, comparing new samples against a curated knowledge base of calibrated prototypes to ensure the final classification is robust.

Experimental Evaluation

The researchers evaluated FedCSPC against nine state-of-the-art (SOTA) methods, including FedAvg, MOON, FedProx, and CCVR. Testing was conducted on four image classification benchmarks:

  • CIFAR10 & CIFAR100: Standard object recognition tasks.
  • TinyImageNet: A high-resolution subset of ImageNet.
  • VireoFood172: A complex dataset featuring high intra-class variance (e.g., diverse presentations of the same food dish).

To simulate real-world heterogeneity, the data was partitioned using a Dirichlet distribution with varying concentration parameters ($\beta=0.1$ for extreme heterogeneity and $\beta=0.5$ for moderate).

Results and Analysis

FedCSPC consistently outperformed existing methods across all tested scenarios. Key findings include:

  • Accuracy Improvements: On the CIFAR10 dataset with high heterogeneity ($\beta=0.1$), FedCSPC achieved an accuracy boost of approximately 3% over FedAvg and surpassed other calibration-focused methods like CCVR.
  • Real-World Robustness: The framework showed significant performance gains on the VireoFood172 dataset, demonstrating its ability to handle the messy, high-variance data typical of real-world "silos."
  • Ablation Studies: Removing either the clustering (DPM) or the augmentation (PA) modules resulted in a drop in performance, confirming that both the diversity of prototypes and the intensity of server-side calibration are critical to the system's success.
  • Compatibility: FedCSPC is designed as a "plug-and-play" enhancement. It can be integrated with existing FL algorithms like FedProx or MOON to provide an immediate boost in accuracy and stability.

Conclusion

FedCSPC represents a significant advancement in addressing the non-IID challenge in Federated Learning. By shifting the focus from weight aggregation to feature space calibration, it allows decentralized systems to build more cohesive and generalized models. Future work will explore extending this prototypical calibration to multimodal tasks and generative models, where data heterogeneity presents even greater challenges.

Summary updated

BibTeX
@inproceedings{qi2023cross,
  author = {Zhuang Qi and Lei Meng and Zitan Chen and Han Hu and Hui Lin and Xiangxu Meng},
  title = {{Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=pgRBEpGsWq},
}
Figure