Introduction
Federated Learning (FL) typically operates under the assumption of homogeneity: all participants utilize identical model architectures and draw from similar data distributions. However, real-world applications often involve diverse hardware capabilities and proprietary model designs, leading to Model Heterogeneity. Furthermore, Data Heterogeneity (or domain shift) occurs when participants observe the same classes (e.g., "digits") through different lenses—such as varying lighting, camera types, or backgrounds.
Beyond heterogeneity, these systems face Catastrophic Forgetting. When a model updates on a small local dataset, it may overfit to that specific data, losing the general knowledge gained from the collective. Conversely, trying to match a global average can cause a model to forget its specialized local expertise. To address these issues, the authors propose FCCL (Federated Cross-Correlation and Continual Learning).
The FCCL Framework
FCCL decouples the learning process into two distinct phases: Collaborative Updating and Local Updating. This "dual-path" approach allows models to learn from the group while maintaining their individual performance.
1. Collaborative Updating: Learning from Others
In a heterogeneous environment, model weights cannot be averaged because architectures differ. FCCL overcomes this by using unlabeled public data as a communication bridge.
Each local model generates logits (predictions) for the public data. The framework then constructs a Cross-Correlation Matrix between these local outputs and the global average. By optimizing this matrix, the system maximizes similarity for identical classes (invariance) while minimizing redundancy between different classes (diversity). This process enables models to learn robust representations that generalize across different domains without requiring identical architectures.
2. Local Updating: Being Yourself
To mitigate catastrophic forgetting during local training on private data, FCCL employs Dual-Domain Knowledge Distillation. The local model is guided by three distinct signals:
- Ground Truth: A standard cross-entropy loss based on private local labels.
- Inter-Domain Teacher: A distillation loss that encourages the model to retain knowledge shared by other participants in previous rounds.
- Intra-Domain Teacher: A distillation loss that anchors the model to its own original, pre-trained state, preserving its specialized "personality."
Experimental Results
The authors evaluated FCCL on image classification benchmarks including Digits (MNIST, USPS, SVHN, Synthetic) and Office-Home (Art, Clipart, Product, Real World).
FCCL consistently outperformed state-of-the-art methods like FedMD and FedDF. On the Digits task, FCCL achieved an average Inter-Domain Accuracy of 33.51%, surpassing FedDF's 30.43%. In terms of Intra-Domain Accuracy, FCCL reached 88.84% on MNIST, significantly higher than the 82.95% achieved by competitors.
Crucially, the method proved robust even when public data was of "low quality" or unrelated (e.g., using Fashion-MNIST to assist in training a Digits classifier). This suggests that FCCL is highly practical for real-world scenarios where perfectly aligned public datasets are unavailable.
Conclusion
FCCL represents a significant step forward for Heterogeneous Federated Learning. By treating logits as a universal language and balancing global collaboration with local preservation through dual-teacher distillation, it effectively navigates the stability-plasticity dilemma. Future work may focus on addressing "Label Skew," where participants hold entirely different categories of data, and further optimizing communication overhead for large-scale public datasets.
Summary updated