Abstract
Distributed Deep Reinforcement Learning (DRL) typically relies on fixed server clusters that incur significant costs due to idle time during synchronization. We present MINIONSRL, the first framework to integrate serverless computing into distributed DRL. By utilizing "pay-as-you-go" serverless functions and an intelligent DRL-based scheduler, MINIONSRL reduces training costs by up to 86% and training time by up to 52% across various standard benchmarks.
Introduction: The Resource Wasting Dilemma
Distributed DRL often follows an actor-learner architecture where multiple "actors" collect environment data and a "learner" uses that data to update the neural network policy. In synchronous on-policy algorithms, a significant bottleneck occurs: actors must wait for the learner to finish its update before starting the next round of data collection.
In traditional server-based environments (e.g., AWS EC2 or Azure VMs), users pay for hardware even when it is idle. Because environment interactions are stochastic, some actors finish faster than others, exacerbating this waste. Since traditional servers take minutes to boot, they cannot be dynamically toggled between rounds, leading to a "resource wasting dilemma" where researchers pay a premium for idle hardware.
Methodology: MINIONSRL
MINIONSRL refactors the DRL training process into independent serverless functions, specifically leveraging Azure Container Instances. This approach addresses the idle-time problem by stopping the "billing meter" the moment an actor finishes its task.
Intelligent Actor Scheduling
The core innovation is a DRL-based scheduler that dynamically determines the optimal number of actors to launch in each round. The scheduler observes the current training state—including learner loss, average rewards, and KL divergence—and makes decisions to minimize total training time while adhering to a strict monetary budget. To mitigate "cold start" latencies inherent in serverless platforms, the framework employs function pre-warming and gRPC for high-speed communication between the learner and distributed actors.
Experimental Evaluation
We evaluated MINIONSRL using six environments from the OpenAI Gym suite, including continuous control tasks (MuJoCo: Hopper, Humanoid, HalfCheetah) and discrete control tasks (Atari: SpaceInvaders, Qbert, Gravitar).
Key Results
MINIONSRL was compared against state-of-the-practice solutions like Azure ML and state-of-the-art architectures like IMPACT.
- Cost Efficiency: The framework achieved a staggering 86% cost reduction in tasks like SpaceInvaders (dropping from ~$26 to ~$7.8).
- Speed: Total training time was reduced by up to 52%. For instance, Humanoid-v3 training time was cut from 464 seconds to 334 seconds.
- Performance: MINIONSRL reached target rewards more reliably than fixed-server baselines, demonstrating that dynamic scaling is superior to static resource allocation.
Discussion and Insights
The bursty, synchronous nature of on-policy DRL is a natural fit for the Function-as-a-Service (FaaS) model. A key insight is that scaling the number of actors during training is highly effective; the scheduler often learns to "burst" actors early for exploration and taper off later to save costs. Furthermore, the scheduler exhibits high transferability; a scheduler trained on one task can be fine-tuned for another in a fraction of the time required for training from scratch, reducing overhead by 90%.
Limitations and Future Work
Despite its success, the initial training of the DRL scheduler is computationally intensive, taking 10-50 hours from scratch. Future work will focus on reducing this overhead and exploring more complex multi-objective optimization strategies to balance wall-clock time, cost, and model accuracy across different cloud providers with varying resource limits.
Summary updated