Abstract
Federated Learning (FL) has traditionally focused on single-task execution, which often leads to significant resource underutilization and data bias in real-world multi-job environments. This paper introduces MJ-FL, a framework designed for parallel multi-job execution in cross-device FL. By employing a "Meta-Greedy" scheduler that synergizes Bayesian Optimization (BODS) and Reinforcement Learning (RLDS), MJ-FL balances hardware efficiency with data fairness. Our evaluation shows that MJ-FL achieves up to 12.7x faster training and 46.4% higher accuracy compared to traditional sequential baselines.
Introduction
Federated Learning (FL) enables training machine learning models on decentralized data without compromising user privacy. However, current research predominantly addresses single-job scenarios. In practice, servers often manage multiple simultaneous tasks—such as training recommendation engines and speech recognizers—using the same pool of heterogeneous user devices. This creates three critical challenges: hardware heterogeneity, resource inefficiency due to device idling, and data fairness issues where prioritizing fast devices biases the model.
The MJ-FL Framework
The proposed Multi-Job Federated Learning (MJ-FL) framework addresses these challenges through a parallel training architecture and a sophisticated cost model. The system architecture, depicted in Figure 1, coordinates the scheduling and aggregation of multiple global models across a shared device pool.
Cost Model and Fairness
MJ-FL utilizes a dynamic cost model that balances Time Cost (minimizing round latency) and Data Fairness Cost (minimizing variance in device selection frequency). By ensuring diverse device participation, the framework prevents model bias and improves global accuracy, particularly in non-IID (Independent and Identically Distributed) data settings.
Scheduling Methodology
The core innovation of MJ-FL is its multi-tiered scheduling approach, which adapts to the complexity of different machine learning tasks:
- BODS (Bayesian Optimization-based Device Scheduling): Uses Gaussian Processes to optimize scheduling for simpler models with fewer parameters.
- RLDS (Reinforcement Learning-based Device Scheduling): Employs an LSTM-based policy network to navigate the complex interference patterns between devices and large-scale models like VGG or ResNet.
- Meta-Greedy Scheduling: This "meta-algorithm" runs six distinct scheduling methods in parallel—including BODS, RLDS, Genetic, and Greedy algorithms—to select the optimal plan for each round based on the current training state.
Experimental Evaluation
The framework was evaluated using standard datasets including Cifar10, MNIST, Fashion-MNIST, and EMNIST, paired with diverse architectures such as VGG16, ResNet18, and AlexNet. Experiments were conducted in both simulated environments and on a real-world cluster of 20 mobile devices on the Baidu AI Cloud.
Key Results
- Efficiency: MJ-FL reached target accuracy up to 12.73 times faster than sequential execution baselines.
- Accuracy: In non-IID environments, the proposed scheduling methods achieved up to 46.4% higher accuracy than simple greedy or random approaches.
- Algorithmic Synergy: The Meta-Greedy approach consistently outperformed standalone RL or BO methods by dynamically shifting focus toward data fairness as training progressed, using a $\sqrt{r}$ scaling factor for fairness weights.
Conclusion and Future Work
MJ-FL demonstrates that parallel multi-job scheduling significantly enhances the efficiency and fairness of cross-device Federated Learning. By treating fairness as a convergence booster rather than a constraint, the framework achieves superior performance in realistic, heterogeneous environments. Future research will explore job prioritization for critical updates and strategies to handle high device churn in volatile networks.
Summary updated