Learn from Others and Be Yourself in Heterogeneous Federated Learning

Wenke Huang, Mang Ye, Bo Du

Introduction

Federated Learning (FL) typically operates under the assumption of homogeneity: all participants utilize identical model architectures and draw from similar data distributions. However, real-world applications often involve diverse hardware capabilities and proprietary model designs, leading to Model Heterogeneity. Furthermore, Data Heterogeneity (or domain shift) occurs when participants observe the same classes (e.g., "digits") through different lenses—such as varying lighting, camera types, or backgrounds.

The challenges of Heterogeneous Federated Learning: Model/Data Heterogeneity and Catastrophic Forgetting.

Beyond heterogeneity, these systems face Catastrophic Forgetting. When a model updates on a small local dataset, it may overfit to that specific data, losing the general knowledge gained from the collective. Conversely, trying to match a global average can cause a model to forget its specialized local expertise. To address these issues, the authors propose FCCL (Federated Cross-Correlation and Continual Learning).

The FCCL Framework

FCCL decouples the learning process into two distinct phases: Collaborative Updating and Local Updating. This "dual-path" approach allows models to learn from the group while maintaining their individual performance.

Overview of the FCCL framework, including (b) Federated Cross-Correlation Learning and (c) Federated Continual Learning.

1. Collaborative Updating: Learning from Others

In a heterogeneous environment, model weights cannot be averaged because architectures differ. FCCL overcomes this by using unlabeled public data as a communication bridge.

Each local model generates logits (predictions) for the public data. The framework then constructs a Cross-Correlation Matrix between these local outputs and the global average. By optimizing this matrix, the system maximizes similarity for identical classes (invariance) while minimizing redundancy between different classes (diversity). This process enables models to learn robust representations that generalize across different domains without requiring identical architectures.

2. Local Updating: Being Yourself

To mitigate catastrophic forgetting during local training on private data, FCCL employs Dual-Domain Knowledge Distillation. The local model is guided by three distinct signals:

  • Ground Truth: A standard cross-entropy loss based on private local labels.
  • Inter-Domain Teacher: A distillation loss that encourages the model to retain knowledge shared by other participants in previous rounds.
  • Intra-Domain Teacher: A distillation loss that anchors the model to its own original, pre-trained state, preserving its specialized "personality."

Experimental Results

The authors evaluated FCCL on image classification benchmarks including Digits (MNIST, USPS, SVHN, Synthetic) and Office-Home (Art, Clipart, Product, Real World).

FCCL consistently outperformed state-of-the-art methods like FedMD and FedDF. On the Digits task, FCCL achieved an average Inter-Domain Accuracy of 33.51%, surpassing FedDF's 30.43%. In terms of Intra-Domain Accuracy, FCCL reached 88.84% on MNIST, significantly higher than the 82.95% achieved by competitors.

Crucially, the method proved robust even when public data was of "low quality" or unrelated (e.g., using Fashion-MNIST to assist in training a Digits classifier). This suggests that FCCL is highly practical for real-world scenarios where perfectly aligned public datasets are unavailable.

Conclusion

FCCL represents a significant step forward for Heterogeneous Federated Learning. By treating logits as a universal language and balancing global collaboration with local preservation through dual-teacher distillation, it effectively navigates the stability-plasticity dilemma. Future work may focus on addressing "Label Skew," where participants hold entirely different categories of data, and further optimizing communication overhead for large-scale public datasets.

Summary updated

Abstract

Federated learning has emerged as an important distributed learning paradigm, which normally involves collaborative updating with others and local updating on private data. However, heterogeneity problem and catastrophic forgetting bring distinctive challenges. First, due to non-i.i.d (identically and independently distributed) data and heterogeneous architectures, models suffer performance degradation on other domains and communication barrier with participants models. Second, in local updating, model is separately optimized on private data, which is prone to overfit current data distribution and forgets previously acquired knowledge, resulting in catastrophic forgetting. In this work, we propose FCCL (Federated CrossCorrelation and Continual Learning). For heterogeneity problem, FCCL leverages unlabeled public data for communication and construct cross-correlation matrix to learn a generalizable representation under domain shift. Mean- while, for catastrophic forgetting, FCCL utilizes knowledge distillation in local updating, providing inter and intra domain information without leaking privacy. Empirical results on various image classification tasks demonstrate the effectiveness of our method and the efficiency of modules.
BibTeX
@inproceedings{huang2022learn,
  author = {Wenke Huang and Mang Ye and Bo Du},
  title = {{Learn from Others and Be Yourself in Heterogeneous Federated Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00990},
}
Figure