VMAgent: Scheduling Simulator for Reinforcement Learning

Junjie Sheng, Shengliang Cai, Haochuan Cui, Wenhao Li, Yun Hua, Bo Jin, Wenli Zhou, Yiqiu Hu, Lei Zhu, Qian Peng, Hongyuan Zha, Xiangfeng WangarXiv 2021 · arXiv

Abstract

Efficient virtual machine (VM) scheduling is critical for cloud infrastructure performance. While reinforcement learning (RL) shows promise for solving these combinatorial optimization problems, the lack of realistic, data-driven simulators has hindered progress. This paper introduces VMAgent, a novel RL simulator built on real-world operational data from Huawei Cloud. VMAgent provides a customizable environment to address challenges such as high-dimensional action spaces and non-stationarity, offering a robust testbed for next-generation resource allocation algorithms.

Introduction

In Infrastructure as a Service (IaaS) environments, the core challenge is the VM scheduling problem: allocating incoming requests for CPU and memory across a massive fleet of servers. Currently, industry standards rely on heuristic methods like "first-fit" or "best-fit." Although efficient, these rule-based approaches often lead to sub-optimal resource layouts and wasted capacity. While Reinforcement Learning (RL) has the potential to outperform these heuristics, existing open-source datasets often lack the necessary context for training accurate agents. VMAgent bridges this gap by providing a realistic sandbox for training and testing RL-based schedulers.

System Architecture and Visualization

VMAgent is designed as a flexible, open-source RL environment. The observation space includes cluster status (server resource occupancy and NUMA architecture) and current request details. The agent's task is to select the optimal server for VM placement.

Key features include:

  • Configuration Flexibility: Simple YAML files define cluster sizes and resource thresholds.
  • Extensive Baselines: Includes standard heuristics and modern RL algorithms (DQN, PPO, SAC, etc.).
  • Rich Visualization Dashboard: A comprehensive interface tracks system health and performance metrics.
The VMAgent dashboard provides real-time visualization of system metrics, including CPU/memory usage, load trends, and heatmaps.

Simulation Scenarios

VMAgent maps practical cloud operations to three core RL challenges:

  1. Recovering Scenario (Non-Stationarity): Simulates a public cloud with constant VM allocation and release, creating a highly dynamic environment.
  2. Fading Scenario (High Dimensionality): Focuses on allocation-only requests in a dedicated cloud, forcing agents to navigate massive action spaces as the cluster grows.
  3. Expansion Scenario (Life-Long Learning): Simulates a growing cloud where new servers are added dynamically, requiring agents to adapt to an expanding environment.

Experimental Evaluation

The simulator is powered by the Huawei-East-1 dataset, containing 241,743 real VM requests. VMAgent's visualization module allows for qualitative and quantitative comparisons between different scheduling strategies.

Comparative analysis between heuristic (best-fit) and RL-based (DQN) scheduling algorithms using VMAgent's comparative visualization tools.

As shown in the comparison, RL agents like DQN can be benchmarked against traditional heuristics to evaluate their impact on cluster load distribution and request success rates.

Conclusion

VMAgent provides a vital infrastructure for the RL community to tackle real-world cloud scheduling problems. Future work will focus on integrating more advanced classical combinatorial optimization methods to provide even more rigorous benchmarks against RL techniques.

Summary updated

Abstract

A novel simulator called VMAgent is introduced to help RL researchers better explore new methods, especially for virtual machine scheduling. VMAgent is inspired by practical virtual machine (VM) scheduling tasks and provides an efficient simulation platform that can reflect the real situations of cloud computing. Three scenarios (fading, recovering, and expansion) are concluded from practical cloud computing and corresponds to many reinforcement learning challenges (high dimensional state and action spaces, high non-stationarity, and life-long demand). VMAgent provides flexible configurations for RL researchers to design their customized scheduling environments considering different problem features. From the VM scheduling perspective, VMAgent also helps to explore better learning-based scheduling solutions.
BibTeX
@misc{sheng2021vmagent,
  author = {Junjie Sheng and Shengliang Cai and Haochuan Cui and Wenhao Li and Yun Hua and Bo Jin and Wenli Zhou and Yiqiu Hu and Lei Zhu and Qian Peng and Hongyuan Zha and Xiangfeng Wang},
  title = {{VMAgent: Scheduling Simulator for Reinforcement Learning}},
  year = {2021},
  eprint = {2112.04785},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.04785},
}
Figure