On-Demand Model and Client Deployment in Federated Learning with Deep Reinforcement Learning

Mario Chahoud, Hani Sami, Azzam Mourad, Hadi Otrok, Jamal Bentahar, Mohsen Guizani

Abstract

In Federated Learning (FL), the limited accessibility of data from diverse locations and user types poses a significant challenge due to restricted user participation. Expanding client access and diversifying data enhance models by incorporating diverse perspectives, thereby enhancing adaptability. However, challenges arise in dynamic and mobile environments where certain devices may become inaccessible as FL clients, impacting data availability and client selection methods. To address this, we propose an On-Demand solution, deploying new clients using Docker Containers on-the-fly. Our On-Demand solution, employing Deep Reinforcement Learning (DRL), targets client availability and selection, while considering data shifts, and container deployment complexities. It employs an autonomous end-to-end solution for handling model deployment and client selection. The DRL strategy uses a Markov Decision Process (MDP) framework, with a Master Learner and a Joiner Learner. The designed cost functions represent the complexity of the dynamic client deployment and selection. Simulated tests show that our architecture can easily adjust to changes in the environment and respond to On-Demand requests. This underscores its ability to improve client availability, capability, accuracy, and learning efficiency, surpassing heuristic and tabular reinforcement learning solutions.

Introduction

Standard Federated Learning (FL) often assumes a stable pool of clients. However, in real-world dynamic environments—such as autonomous vehicle fleets or wearable health monitor networks—devices frequently move in and out of range. This instability leads to two primary issues: data scarcity and bias from missing mobile device perspectives, and a "software gap" where devices with relevant data lack the necessary libraries or APIs for training. Existing solutions using simple heuristics or random selection often fail to adapt to rapid data distribution shifts (non-IID data).

The On-Demand Framework

The proposed framework treats client participation as a deployment challenge. Instead of waiting for a device to be ready, the server pushes a lightweight Docker container to volunteer devices, encapsulating the model, training methods, and APIs.

System Architecture

The architecture is structured into three main layers: the Server, the Orchestrator, and the Volunteering Device. This hierarchy allows for efficient management of resources and intelligent client selection.

The proposed On-Demand Federated Learning architecture featuring Server, Orchestrator, and Volunteering Device layers.

A dual-stage Deep Reinforcement Learning (DRL) strategy using Deep Q-Networks (DQN) manages this complexity. A Master Learner trains offline on historical data to establish a base model, while a Joiner Learner refines the strategy online to handle real-time environmental changes. The agent balances goals such as minimizing active hosts, maximizing data diversity, and adhering to physical hardware constraints like RAM and battery capacity.

Algorithmic Approach

The training process follows a structured round-based approach where clients are dynamically selected and deployed based on the DRL agent's decisions.

The On-Demand FL Algorithm detailing the round-based training and deployment process.

Experimental Evaluation

The framework was evaluated using the Mobile Data Challenge (MDC) dataset, which contains real-world movement records from 47 users. This dataset simulates realistic data shifts as users move between different environments, providing a rigorous test for the DRL agent's adaptability.

Key Results

  • Rapid Convergence: The DRL approach reached target accuracy in under 10 rounds, significantly faster than tabular RL or heuristic methods.
  • High Efficiency: The framework achieved a 92% acceptance rate for training rounds, compared to random selection which discarded over 60% of rounds due to poor client quality.
  • Adaptability: The DRL agent successfully re-prioritized deployments during simulated data shifts, maintaining model performance where heuristic models saw sharp drops in accuracy.

Conclusion

By treating client mobility as an opportunity for proactive data scouting rather than a failure, the On-Demand FL framework significantly improves model adaptability and training efficiency. Future work will address security hardening for orchestrator nodes and explore multi-model deployment strategies to further optimize edge resource utilization.

Summary updated

Abstract

In Federated Learning (FL), the limited accessibility of data from diverse locations and user types poses a significant challenge due to restricted user participation. Expanding client access and diversifying data enhance models by incorporating diverse perspectives, thereby enhancing adaptability. However, challenges arise in dynamic and mobile environments where certain devices may become inaccessible as FL clients, impacting data availability and client selection methods. To address this, we propose an On-Demand solution, deploying new clients using Docker Containers on-the-fly. Our On-Demand solution, employing Deep Reinforcement Learning (DRL), targets client availability and selection, while considering data shifts, and container deployment complexities. It employs an autonomous end-to-end solution for handling model deployment and client selection. The DRL strategy uses a Markov Decision Process (MDP) framework, with a Master Learner and a Joiner Learner. The designed cost functions represent the complexity of the dynamic client deployment and selection. Simulated tests show that our architecture can easily adjust to changes in the environment and respond to On-Demand requests. This underscores its ability to improve client availability, capability, accuracy, and learning efficiency, surpassing heuristic and tabular reinforcement learning solutions.
BibTeX
@article{chahoud2025demand,
  author = {Mario Chahoud and Hani Sami and Azzam Mourad and Hadi Otrok and Jamal Bentahar and Mohsen Guizani},
  title = {{On-Demand Model and Client Deployment in Federated Learning with Deep Reinforcement Learning}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2025.3561722},
}
Figure