On-Demand Model and Client Deployment in Federated Learning With Deep Reinforcement Learning

Mario Chahoud, Hani Sami, Azzam Mourad, Hadi Otrok, Jamal Bentahar, Mohsen Guizani

This paper, recently published by researchers from Concordia University and MBZUAI, tackles one of the most persistent "real-world" headaches in Federated Learning (FL): the fact that clients aren't static, reliable, or always ready to train.

TL;DR

The authors propose an "On-Demand" Federated Learning framework that uses Docker containers to deploy training environments to mobile devices on-the-fly. By employing a dual-stage Deep Reinforcement Learning (DRL) strategy, the system intelligently decides where to deploy clients to maximize data diversity and minimize resource waste, reaching target accuracy in fewer than 10 rounds.

What Problem Does This Paper Address?

Standard Federated Learning assumes a relatively stable pool of clients. However, in dynamic environments—like a fleet of autonomous vehicles or a network of wearable health monitors—devices are constantly moving in and out of range.

This creates two major issues:

  1. Data Scarcity/Bias: If you only train on "static" clients, your model misses out on the diverse data held by mobile devices, leading to a biased global model.
  2. The "Software Gap": A device might have great data but lack the specific libraries or APIs needed to participate in a training round.

Existing solutions often rely on simple heuristics or "Vanilla" random selection, which struggle to adapt when data distributions shift rapidly (a phenomenon known as non-IID data).

Methods

The core innovation is the On-Demand solution, which treats client participation as a deployment problem rather than just a selection problem.

  • Containerization (Docker): Instead of waiting for a device to be "ready," the server pushes a lightweight Docker container to volunteer devices. This container encapsulates everything needed: the model, the training methods, and the APIs.
  • Deep Reinforcement Learning (DQN): Because the "state" of a mobile network is too complex for traditional math, the authors use a Deep Q-Network. The agent learns to balance four conflicting goals:
  • Minimizing the number of active hosts (saving energy).
  • Maximizing data volume and diversity.
  • Prioritizing high-accuracy clients.
  • Fulfilling requests from local "orchestrators" in high-activity areas.
  • Master & Joiner Learner Architecture: To solve the "cold start" problem where DRL takes too long to learn, they split the brain. A Master Learner trains offline on historical data to create a mature base model. A Joiner Learner then takes over online, refining the strategy in real-time to handle sudden environmental changes.
  • Kubeadm Orchestration: They use Kubernetes tools to manage these clusters of edge devices, ensuring that if a "Master" node in a local cluster fails, another is elected immediately.

Datasets

The researchers evaluated their framework using the Mobile Data Challenge (MDC) dataset.

  • Context: This dataset contains real-world movement records from 47 users, including GPS coordinates and time indicators.
  • Why it matters: It allows the simulation of realistic "data shifts." As users move between home, work, and social areas, the type of data they generate changes, providing a rigorous test for the DRL agent's ability to adapt.

Key Insights and Ideas

  • Mobility as an Opportunity: Most FL papers treat client mobility as a "failure" (a dropped connection). This paper treats it as a way to "scout" for better data. If a specific geographic area is underrepresented in the model, the DRL agent proactively deploys containers to devices entering that area.
  • The "Pun" Factor: The authors implemented a punitive cost factor in their MDP (Markov Decision Process). If the DRL agent tries to deploy a model that exceeds a device's RAM or battery capacity, it receives a heavy penalty, forcing it to learn the physical constraints of edge hardware.
  • Efficiency over Randomness: In many FL setups, a round is "discarded" if the data is poor. This framework achieved a 92% acceptance rate for training rounds, whereas random selection discarded over 60% of rounds due to poor client quality.

Key Results

  • Rapid Convergence: The DRL-based approach reached target accuracy in under 10 rounds. For comparison, traditional tabular RL and heuristic methods took significantly longer or failed to reach the same stability.
  • Superior Accuracy: The On-Demand deployment outperformed "Vanilla FL" (random selection) by effectively "hunting" for diverse data rather than waiting for it to appear.
  • Adaptability: When simulated data shifts occurred (representing clients moving to new locations), the DRL agent successfully re-prioritized deployments to maintain model performance, while heuristic models saw sharp drops in accuracy.

Limitations and Future Work

  • Security Concerns: The authors briefly acknowledge that allowing a central server to deploy containers onto mini-servers (orchestrators) raises security and trust issues. This "confidence" in the mini-servers is assumed but not technically hardened in this paper.
  • Single Model Focus: The current architecture is optimized for one global model. The authors suggest that future work should explore multi-model deployment, where the DRL agent must decide which type of model (e.g., traffic prediction vs. weather) to deploy to which device.
  • Reward Shaping: While the current cost function is effective, the authors plan to explore more complex "reward shaping" techniques to further fine-tune the balance between energy consumption and model precision.

Summary updated

Abstract

In Federated Learning (FL), the limited accessibility of data from diverse locations and user types poses a significant challenge due to restricted user participation. Expanding client access and diversifying data enhance models by incorporating diverse perspectives, thereby enhancing adaptability. However, challenges arise in dynamic and mobile environments where certain devices may become inaccessible as FL clients, impacting data availability and client selection methods. To address this, we propose an On-Demand solution, deploying new clients using Docker Containers on-the-fly. Our On-Demand solution, employing Deep Reinforcement Learning (DRL), targets client availability and selection, while considering data shifts, and container deployment complexities. It employs an autonomous end-to-end solution for handling model deployment and client selection. The DRL strategy uses a Markov Decision Process (MDP) framework, with a Master Learner and a Joiner Learner. The designed cost functions represent the complexity of the dynamic client deployment and selection. Simulated tests show that our architecture can easily adjust to changes in the environment and respond to On-Demand requests. This underscores its ability to improve client availability, capability, accuracy, and learning efficiency, surpassing heuristic and tabular reinforcement learning solutions.
BibTeX
@article{chahoud2025demand,
  author = {Mario Chahoud and Hani Sami and Azzam Mourad and Hadi Otrok and Jamal Bentahar and Mohsen Guizani},
  title = {{On-Demand Model and Client Deployment in Federated Learning With Deep Reinforcement Learning}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2025.3561722},
}

Loading related content...

Figure