Energy-Efficient Container Scheduling Based on Deep Reinforcement Learning in Data Centers

Zhuohui Li, Shaofeng Zhang, Yiqian Li, Xingchen Liu, Junyang Huang, Jinlong HuComput 2025 · DOI

TL;DR

This paper introduces SAC-CS, a novel container scheduling algorithm that uses deep reinforcement learning (DRL) to simultaneously optimize for task execution time and energy consumption in data centers. By leveraging a maximum entropy framework, SAC-CS achieves a better balance between these competing objectives than existing heuristic and DRL methods, leading to significant improvements in efficiency and energy savings.

What Problem Does This Paper Address?

Modern data centers are the backbone of cloud computing, AI, and big data. As they grow, managing their efficiency becomes critical. Traditional container scheduling methods often struggle with the sheer volume and diversity of tasks and resources. They typically focus on either minimizing task completion time or reducing energy consumption, but rarely both effectively. This leads to a trade-off where optimizing one often degrades the other, resulting in inefficient resource utilization and high energy bills. The challenge is to find a scheduling strategy that can dynamically adapt to changing workloads and resource heterogeneity to achieve a harmonious balance between performance and energy efficiency.

Methods

The core of this paper is the SAC-CS (Soft Actor-Critic based Container Scheduling) algorithm. It's built upon the Soft Actor-Critic (SAC) framework, a state-of-the-art deep reinforcement learning approach. Here's what makes it special:

  • Maximum Entropy Reinforcement Learning: Unlike traditional RL that solely focuses on maximizing rewards, SAC also maximizes the entropy of the policy. This encourages exploration, preventing the agent from getting stuck in suboptimal solutions and leading to more robust and adaptable policies.
  • Actor-Critic Architecture: SAC-CS uses a dual-network system:
  • The Actor network decides which host to assign a task to (the action).
  • The Critic network evaluates the quality of these decisions (the Q-value). This allows for simultaneous optimization of the scheduling policy and the value function.

  • Stochastic Policy: Instead of making deterministic decisions, SAC-CS outputs a probability distribution over possible actions. This allows for more nuanced and adaptive scheduling, especially in dynamic environments.

  • Multi-Objective Reward Function: The reward signal is carefully designed to balance two key objectives: task runtime and host energy consumption. This is achieved through a weighted sum, allowing for flexible trade-offs.

Datasets

The paper evaluates SAC-CS using two types of datasets:

  • Synthetic Workloads: This dataset is derived from the Alibaba Cluster Trace Program. It simulates a two-level dependency relationship between tasks and includes parameters like the number of tasks, containers, and container runtime.
  • Alibaba Cluster Workloads: This dataset uses the Cluster Trace GPU v2025 dataset from Alibaba, focusing on deep learning recommendation model inference services. It's characterized by latency-sensitive tasks and provides multi-dimensional resource data (CPU, GPU, memory, etc.). This dataset was preprocessed to align with the experimental setup.

Key Insights and Ideas

  • DRL Outperforms Heuristics: The study strongly demonstrates that Deep Reinforcement Learning (DRL) algorithms, including SAC-CS, DQN, and PPO, significantly outperform traditional heuristic scheduling methods across various metrics. This highlights DRL's ability to learn complex dependencies and adapt to dynamic environments.
  • Maximum Entropy is Key for Balance: The "maximum entropy" aspect of SAC-CS is crucial. It encourages exploration, preventing the agent from over-converging on a single "best" host (like the fastest CPU) and instead promoting a more balanced distribution of tasks across available resources. This is vital for avoiding bottlenecks and achieving true multi-objective optimization.
  • Stochastic Policies are More Robust: The paper suggests that stochastic policies are inherently better at managing the burstiness of task arrivals and resource contention in data centers compared to deterministic policies.
  • PerformanceFirst's Pitfall: The heuristic "PerformanceFirst" algorithm, while good at reducing task runtime, fails to consider energy consumption, leading to poor energy efficiency. This underscores the need for algorithms that explicitly balance both objectives.
  • Network Load Impact is Minimal for SAC-CS: Even under varying network loads, SAC-CS maintains robust performance, indicating its strong adaptability to environmental changes.

Key Results

  • Time-Energy Metric (F): SAC-CS consistently achieved the lowest Time-Energy product (F) across both synthetic and Alibaba cluster workloads, indicating the best balance between task execution time and energy consumption. For synthetic workloads, SAC-CS achieved F=0.057, significantly better than DQN (0.065) and PPO (0.066). For Alibaba workloads, SAC-CS achieved F=0.264, compared to DQN (0.272) and PPO (0.276).
  • Reduced Waiting and Runtime: SAC-CS significantly reduced average container waiting times and runtimes compared to both heuristic and other DRL methods. For instance, on Alibaba workloads, SAC-CS reduced waiting time by 23% to 35% compared to the simulation environment.
  • Intelligent Load Balancing: Figure 5 shows SAC-CS distributing tasks more evenly across hosts of different computing speeds, avoiding the congestion seen with heuristics like PerformanceFirst and the over-convergence of DQN/PPO on the fastest hosts.

Limitations and Future Work

  • Simulation Environment: The current evaluation is based on a simulation environment. The authors acknowledge that real-world scenarios might introduce more complex dynamics.
  • Linear Energy Model: The study assumes a linear energy consumption model for hosts, which might not accurately reflect non-linear energy behaviors in real data centers.
  • Initial Training Instability: The algorithm might experience instability during the initial "cold start" training phase.

Future work includes:

  • Deploying SAC-CS in a more comprehensive simulation environment and on a physical Kubernetes testbed for rigorous evaluation.
  • Integrating thermal-aware objectives into the reward function.
  • Exploring transfer learning techniques to speed up model adaptation across different hardware configurations.

Summary updated

BibTeX
@article{li2025energy,
  author = {Zhuohui Li and Shaofeng Zhang and Yiqian Li and Xingchen Liu and Junyang Huang and Jinlong Hu},
  title = {{Energy-Efficient Container Scheduling Based on Deep Reinforcement Learning in Data Centers}},
  journal = {Comput.},
  year = {2025},
  doi = {10.3390/computers14120560},
}
Figure