Federated Model Heterogeneous Matryoshka Representation Learning

Liping Yi, Han Yu, Chao Ren, Gang Wang, xiaoguang Liu, Xiaoxiao Li

Abstract

Federated Learning (FL) traditionally requires all participating clients to share a uniform model architecture. However, real-world deployments often face significant system, data, and model heterogeneity. This paper introduces FedMRL, a novel framework designed to enable collaboration between clients with entirely different model architectures. By combining a shared auxiliary small model with Multi-Granularity Matryoshka Representation Learning (MRL), FedMRL achieves superior accuracy and communication efficiency without requiring shared public datasets or compromising the privacy of local "black-box" models.

1. Introduction

Standard Federated Learning (FL) protocols, such as FedAvg, operate under the assumption that all clients utilize an identical model architecture. In practice, this assumption is frequently violated due to three primary factors:

  1. System Heterogeneity: Hardware capabilities vary from high-performance servers to low-power IoT devices.
  2. Data Heterogeneity: Local data distributions are often non-independently and identically distributed (Non-IID), complicating global convergence.
  3. Model Heterogeneity: Organizations may use proprietary architectures that they are unwilling to disclose or standardize.

Existing approaches to Model-Heterogeneous FL (MHeteroFL) typically rely on Knowledge Distillation (KD). While KD allows models to learn from one another's outputs, it often requires a shared public dataset and suffers from high communication overhead and limited knowledge transfer. FedMRL addresses these gaps by shifting the focus from output mimicking to representation fusion.

2. The FedMRL Framework

FedMRL introduces two core innovations to bridge the gap between diverse local models: Adaptive Representation Fusion and Multi-Granularity Matryoshka Representation Learning.

2.1 Adaptive Representation Fusion

In the FedMRL architecture, each client maintains their unique, proprietary local model alongside a shared, auxiliary "homogeneous" small model provided by the server. Instead of forcing the local model to mimic the global model's predictions, FedMRL extracts features from both.

These features are "spliced" together and passed through a lightweight, personalized Representation Projector. This projector maps the combined data into a fused representation that is specifically adapted to the client's local data distribution. This "splicing" approach preserves the semantic integrity of both generalized global information and personalized local features.

2.2 Multi-Granularity Matryoshka Representation Learning (MRL)

Inspired by the structure of Russian nesting dolls, FedMRL utilizes Matryoshka Representation Learning to create embeddings at multiple scales. The fused representation is used to generate two distinct levels of granularity:

  • Coarse-Granularity: A low-dimensional representation handled by the global model's header, focusing on high-level semantic features.
  • Fine-Granularity: A high-dimensional representation handled by the local heterogeneous model's header, capturing intricate details specific to the local dataset.

By training on both granularities simultaneously, the system learns from multiple perspectives. Crucially, only the parameters of the small auxiliary model are transmitted to the server, ensuring the privacy of the local model's internal architecture.

3. Theoretical Foundations

Beyond empirical performance, the authors provide a formal proof of the framework's stability. FedMRL is shown to achieve an $O(1/T)$ non-convex convergence rate. This mathematical guarantee ensures that the system is robust and will stabilize over time, even when dealing with highly heterogeneous client environments.

4. Experimental Evaluation

4.1 Setup

The framework was evaluated using the CIFAR-10 and CIFAR-100 image classification benchmarks. To simulate real-world complexity, two Non-IID settings were employed:

  • Class-based: Clients are restricted to a subset of classes.
  • Dirichlet-based: A Dirichlet distribution ($\alpha$) is used to create unbalanced data ratios across the client pool.

4.2 Performance Results

FedMRL was benchmarked against seven state-of-the-art MHeteroFL methods, including FedProto, FML, and FedKD. The results demonstrate significant advantages:

  • Accuracy Gains: FedMRL achieved an average accuracy improvement of 8.48% over the best overall baseline. Compared to "same-category" mutual learning baselines, the improvement reached as high as 24.94%.
  • Client Personalization: In a 100-client simulation, 99% of participants performed better using FedMRL than any competing method, highlighting its adaptability to local data.
  • Efficiency: Despite the addition of an auxiliary model, FedMRL converges faster than its peers. This leads to lower total computational costs (FLOPs) and fewer communication rounds to reach target accuracy, making it ideal for bandwidth-constrained environments.

5. Conclusion and Future Work

FedMRL represents a significant step forward in making Federated Learning viable for heterogeneous ecosystems. By leveraging multi-granular representations and feature splicing, it overcomes the limitations of traditional distillation-based methods.

Future research will focus on MRL-E, an optimized version of Matryoshka learning designed to remove the global header entirely. This would further reduce the storage and computational footprint of the framework while maintaining its multi-perspective learning benefits. Additionally, the team aims to refine inference-time configurations to better suit specific hardware constraints.

Summary updated

Abstract

Model heterogeneous federated learning (MHeteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. However, existing MHeteroFL methods rely on training loss to transfer knowledge between the client model and the server model, resulting in limited knowledge exchange. To address this limitation, we propose the **Fed**erated model heterogeneous **M**atryoshka **R**epresentation **L**earning (**FedMRL**) approach for supervised learning tasks. It adds an auxiliary small homogeneous model shared by clients with heterogeneous local models. (1) The generalized and personalized representations extracted by the two models' feature extractors are fused by a personalized lightweight representation projector. This step enables representation fusion to adapt to local data distribution. (2) The fused representation is then used to construct Matryoshka representations with multi-dimensional and multi-granular embedded representations learned by the global homogeneous model header and the local heterogeneous model header. This step facilitates multi-perspective representation learning and improves model learning capability. Theoretical analysis shows that FedMRL achieves a $O(1/T)$ non-convex convergence rate. Extensive experiments on benchmark datasets demonstrate its superior model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art and the best same-category baseline, respectively.
BibTeX
@inproceedings{yi2024federated,
  author = {Liping Yi and Han Yu and Chao Ren and Gang Wang and xiaoguang Liu and Xiaoxiao Li},
  title = {{Federated Model Heterogeneous Matryoshka Representation Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=5yboFMpvHf},
}
Figure