FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices

Dezhong Yao, Yuexin Shi, Tongtong Liu, Zhiqiang XuarXiv 2025 · DOI · arXiv

Abstract

Federated Learning (FL) typically involves multiple communication rounds, posing challenges for resource-constrained edge devices. One-shot FL addresses this but often assumes model homogeneity, which is unrealistic in diverse edge networks. We present FedMHO (Model-Heterogeneous One-shot FL), a framework that allows capable devices to train deep classifiers while constrained devices train lightweight generative models. By leveraging synthetic data generation and knowledge distillation, FedMHO achieves high accuracy in a single round without auxiliary datasets, outperforming existing baselines by up to 8.35%.

Introduction

Standard Federated Learning is an iterative process where devices train locally and send updates to a central server over hundreds of rounds. This creates a massive communication bottleneck and drains the battery of edge devices. While "one-shot" FL reduces this to a single round of communication, it usually assumes every device runs the exact same model architecture.

In reality, edge networks are highly heterogeneous. A smartphone may have gigabytes of RAM, while a smart wristband might have less than 2MB. Forcing a uniform model size either wastes the potential of powerful devices or excludes resource-strapped ones. Previous attempts to mix model sizes often suffered from "negative gain," where poor performance from small models degraded the final global model.

Methodology

FedMHO addresses hardware heterogeneity by splitting clients into two groups:

  1. Resource-Sufficient Clients: Train deep classification models (e.g., VGG-9).
  2. Resource-Constrained Clients: Train lightweight Conditional Variational Autoencoders (CVAEs). Instead of learning to classify, these devices learn to generate synthetic versions of their local data.
The FedMHO framework architecture, illustrating the interaction between heterogeneous clients and the server-side knowledge fusion process.

Once the server receives these models, it follows a two-stage process:

  • Data Generation & Optimization: The server uses the CVAE decoders to generate synthetic samples. To ensure quality, an unsupervised data optimization strategy clusters the generated images using K-means and filters out "noisy" or mislabeled samples that fall too far from the cluster centers.
  • Knowledge Fusion: The server initializes the global model by averaging the parameters of the deep classifiers (FedAvg). It then fine-tunes this model using the optimized synthetic data.

To prevent the model from "forgetting" knowledge learned from the classifiers while learning from synthetic data, the authors propose two variants:

  • FedMHO-MD (Multi-teacher Distillation): Uses the original local classifiers as "teachers" to guide the global model.
  • FedMHO-SD (Self-Distillation): Uses the initial averaged global model as a teacher to guide its own subsequent training.

Evaluation and Key Results

The framework was evaluated on four standard computer vision benchmarks (MNIST, Fashion-MNIST, SVHN, and EMNIST), partitioned using a Dirichlet distribution ($\alpha$) to simulate non-IID (statistically diverse) data across clients.

Top-1 Test Accuracy comparison between FedMHO and state-of-the-art baselines across different levels of data heterogeneity.

FedMHO consistently outperformed state-of-the-art one-shot FL baselines, including DENSE, Co-Boosting, and FEDCVAE:

  • Accuracy Boost: Compared to the best-performing baseline, FedMHO-MD/SD improved average accuracy by 5.17% to 8.35%.
  • Robustness to Heterogeneity: In highly non-IID settings (Dir(0.1)), FedMHO maintained a significant lead, whereas traditional methods like FedAvg saw performance collapse.
  • Efficiency: FedMHO reaches target accuracy levels in significantly fewer server-side epochs than baselines, making the central aggregation process much faster.

Discussion and Insights

A key finding of this research is the relative impact of weak classifiers versus weak generators. While a tiny classifier on a weak device provides "erroneous logits" (misleading predictions) that hurt the global model, a tiny generator merely produces "blurry" images. Blurry images are less damaging to the fusion process than confident, incorrect predictions.

Furthermore, the "Knowledge-Forgetting" trap is a significant hurdle in one-shot FL. Simply training a global model on synthetic data after initializing it with local weights causes the model to drift away from the high-quality knowledge in the original classifiers. Distillation (MD or SD) acts as a crucial regularizer to anchor the model.

Limitations and Future Work

Despite its success, FedMHO has limitations. While synthetic data helps protect raw data, one-shot FL remains vulnerable to certain types of reconstruction or membership inference attacks. Exploring these vulnerabilities is a primary direction for future work. Additionally, the current study focuses on VGG and CNN architectures; the framework's performance on much larger Transformer-based models in an edge context remains an open question. Finally, the server requires knowledge of local label distributions to generate the correct ratio of synthetic samples, representing a small but notable piece of shared metadata.

Summary updated

BibTeX
@misc{yao2025fedmho,
  author = {Dezhong Yao and Yuexin Shi and Tongtong Liu and Zhiqiang Xu},
  title = {{FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices}},
  year = {2025},
  eprint = {2502.08518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.08518},
}
Figure