Abstract
Federated Learning (FL) typically involves multiple communication rounds, posing challenges for resource-constrained edge devices. One-shot FL addresses this but often assumes model homogeneity, which is unrealistic in diverse edge networks. We present FedMHO (Model-Heterogeneous One-shot FL), a framework that allows capable devices to train deep classifiers while constrained devices train lightweight generative models. By leveraging synthetic data generation and knowledge distillation, FedMHO achieves high accuracy in a single round without auxiliary datasets, outperforming existing baselines by up to 8.35%.
Introduction
Standard Federated Learning is an iterative process where devices train locally and send updates to a central server over hundreds of rounds. This creates a massive communication bottleneck and drains the battery of edge devices. While "one-shot" FL reduces this to a single round of communication, it usually assumes every device runs the exact same model architecture.
In reality, edge networks are highly heterogeneous. A smartphone may have gigabytes of RAM, while a smart wristband might have less than 2MB. Forcing a uniform model size either wastes the potential of powerful devices or excludes resource-strapped ones. Previous attempts to mix model sizes often suffered from "negative gain," where poor performance from small models degraded the final global model.
Methodology
FedMHO addresses hardware heterogeneity by splitting clients into two groups:
- Resource-Sufficient Clients: Train deep classification models (e.g., VGG-9).
- Resource-Constrained Clients: Train lightweight Conditional Variational Autoencoders (CVAEs). Instead of learning to classify, these devices learn to generate synthetic versions of their local data.
Once the server receives these models, it follows a two-stage process:
- Data Generation & Optimization: The server uses the CVAE decoders to generate synthetic samples. To ensure quality, an unsupervised data optimization strategy clusters the generated images using K-means and filters out "noisy" or mislabeled samples that fall too far from the cluster centers.
- Knowledge Fusion: The server initializes the global model by averaging the parameters of the deep classifiers (FedAvg). It then fine-tunes this model using the optimized synthetic data.
To prevent the model from "forgetting" knowledge learned from the classifiers while learning from synthetic data, the authors propose two variants:
- FedMHO-MD (Multi-teacher Distillation): Uses the original local classifiers as "teachers" to guide the global model.
- FedMHO-SD (Self-Distillation): Uses the initial averaged global model as a teacher to guide its own subsequent training.
Evaluation and Key Results
The framework was evaluated on four standard computer vision benchmarks (MNIST, Fashion-MNIST, SVHN, and EMNIST), partitioned using a Dirichlet distribution ($\alpha$) to simulate non-IID (statistically diverse) data across clients.
FedMHO consistently outperformed state-of-the-art one-shot FL baselines, including DENSE, Co-Boosting, and FEDCVAE:
- Accuracy Boost: Compared to the best-performing baseline, FedMHO-MD/SD improved average accuracy by 5.17% to 8.35%.
- Robustness to Heterogeneity: In highly non-IID settings (Dir(0.1)), FedMHO maintained a significant lead, whereas traditional methods like FedAvg saw performance collapse.
- Efficiency: FedMHO reaches target accuracy levels in significantly fewer server-side epochs than baselines, making the central aggregation process much faster.
Discussion and Insights
A key finding of this research is the relative impact of weak classifiers versus weak generators. While a tiny classifier on a weak device provides "erroneous logits" (misleading predictions) that hurt the global model, a tiny generator merely produces "blurry" images. Blurry images are less damaging to the fusion process than confident, incorrect predictions.
Furthermore, the "Knowledge-Forgetting" trap is a significant hurdle in one-shot FL. Simply training a global model on synthetic data after initializing it with local weights causes the model to drift away from the high-quality knowledge in the original classifiers. Distillation (MD or SD) acts as a crucial regularizer to anchor the model.
Limitations and Future Work
Despite its success, FedMHO has limitations. While synthetic data helps protect raw data, one-shot FL remains vulnerable to certain types of reconstruction or membership inference attacks. Exploring these vulnerabilities is a primary direction for future work. Additionally, the current study focuses on VGG and CNN architectures; the framework's performance on much larger Transformer-based models in an edge context remains an open question. Finally, the server requires knowledge of local label distributions to generate the correct ratio of synthetic samples, representing a small but notable piece of shared metadata.
Summary updated