Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing

Hanfei Yu, Jian Li, Yang Hua, Xu Yuan, Hao Wang

Abstract

Distributed Deep Reinforcement Learning (DRL) typically relies on fixed server clusters that incur significant costs due to idle time during synchronization. We present MINIONSRL, the first framework to integrate serverless computing into distributed DRL. By utilizing "pay-as-you-go" serverless functions and an intelligent DRL-based scheduler, MINIONSRL reduces training costs by up to 86% and training time by up to 52% across various standard benchmarks.

Introduction: The Resource Wasting Dilemma

Distributed DRL often follows an actor-learner architecture where multiple "actors" collect environment data and a "learner" uses that data to update the neural network policy. In synchronous on-policy algorithms, a significant bottleneck occurs: actors must wait for the learner to finish its update before starting the next round of data collection.

In traditional server-based environments (e.g., AWS EC2 or Azure VMs), users pay for hardware even when it is idle. Because environment interactions are stochastic, some actors finish faster than others, exacerbating this waste. Since traditional servers take minutes to boot, they cannot be dynamically toggled between rounds, leading to a "resource wasting dilemma" where researchers pay a premium for idle hardware.

Comparison of server-based vs. serverless training architectures. Serverless functions release resources immediately upon task completion, eliminating idle costs.

Methodology: MINIONSRL

MINIONSRL refactors the DRL training process into independent serverless functions, specifically leveraging Azure Container Instances. This approach addresses the idle-time problem by stopping the "billing meter" the moment an actor finishes its task.

Intelligent Actor Scheduling

The core innovation is a DRL-based scheduler that dynamically determines the optimal number of actors to launch in each round. The scheduler observes the current training state—including learner loss, average rewards, and KL divergence—and makes decisions to minimize total training time while adhering to a strict monetary budget. To mitigate "cold start" latencies inherent in serverless platforms, the framework employs function pre-warming and gRPC for high-speed communication between the learner and distributed actors.

Experimental Evaluation

We evaluated MINIONSRL using six environments from the OpenAI Gym suite, including continuous control tasks (MuJoCo: Hopper, Humanoid, HalfCheetah) and discrete control tasks (Atari: SpaceInvaders, Qbert, Gravitar).

Key Results

MINIONSRL was compared against state-of-the-practice solutions like Azure ML and state-of-the-art architectures like IMPACT.

Performance comparison of MINIONSRL against Azure ML and IMPACT across various OpenAI Gym environments. MINIONSRL consistently reaches target rewards faster in wall-clock time.
  • Cost Efficiency: The framework achieved a staggering 86% cost reduction in tasks like SpaceInvaders (dropping from ~$26 to ~$7.8).
  • Speed: Total training time was reduced by up to 52%. For instance, Humanoid-v3 training time was cut from 464 seconds to 334 seconds.
  • Performance: MINIONSRL reached target rewards more reliably than fixed-server baselines, demonstrating that dynamic scaling is superior to static resource allocation.

Discussion and Insights

The bursty, synchronous nature of on-policy DRL is a natural fit for the Function-as-a-Service (FaaS) model. A key insight is that scaling the number of actors during training is highly effective; the scheduler often learns to "burst" actors early for exploration and taper off later to save costs. Furthermore, the scheduler exhibits high transferability; a scheduler trained on one task can be fine-tuned for another in a fraction of the time required for training from scratch, reducing overhead by 90%.

Limitations and Future Work

Despite its success, the initial training of the DRL scheduler is computationally intensive, taking 10-50 hours from scratch. Future work will focus on reducing this overhead and exploring more complex multi-objective optimization strategies to balance wall-clock time, cost, and model accuracy across different cloud providers with varying resource limits.

Summary updated

Abstract

Deep reinforcement learning (DRL) has gained immense success in many applications, including gaming AI, robotics, and system scheduling. Distributed algorithms and architectures have been vastly proposed (e.g., actor-learner architecture) to accelerate DRL training with large-scale server-based clusters. However, training on-policy algorithms with the actor-learner architecture unavoidably induces resource wasting due to synchronization between learners and actors, thus resulting in significantly extra billing. As a promising alternative, serverless computing naturally fits on-policy synchronization and alleviates resource wasting in distributed DRL training with pay-as-you-go pricing. Yet, none has leveraged serverless computing to facilitate DRL training. This paper proposes MinionsRL, the first serverless distributed DRL training framework that aims to accelerate DRL training- and cost-efficiency with dynamic actor scaling. We prototype MinionsRL on top of Microsoft Azure Container Instances and evaluate it with popular DRL tasks from OpenAI Gym. Extensive experiments show that MinionsRL reduces total training time by up to 52% and training cost by 86% compared to latest solutions.
BibTeX
@inproceedings{yu2024cheaper,
  author = {Hanfei Yu and Jian Li and Yang Hua and Xu Yuan and Hao Wang},
  title = {{Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29592},
}
Figure