Abstract
Efficient resource scheduling in Computing Power Networks (CPN) is very important for improving system performance and reducing energy consumption, which are two key factors for the development of low-consumption computing technologies. To address the challenges posed by sparse rewards in the CPN, we propose CPN Resource Scheduling with Reward Shaping (C2RS), which is a novel scheduler developed using Deep Reinforcement Learning (DRL). By incorporating a customized reward shaping mechanism, C2RS infers a dense reward function that is closely consistent with the original sparse reward signal. This approach is able to learn optimal scheduling policies faster and significantly improves the efficiency of baseline methods. Our comprehensive evaluation in the simulated CPN environment shows that C2RS outperforms existing DRL-based schedulers on multiple performance metrics. Importantly, C2RS not only improves learning efficiency, but also leads to the development of more effective scheduling policies through its innovative reward shaping, thereby contributing to the advancement of energy-efficient computing solutions for big data analytics.
Introduction
In the era of big data, the paradigm is shifting from centralized data centers toward Computing Power Networks (CPNs). These decentralized, geographically distributed networks treat computing resources—such as CPU, memory, and bandwidth—as a utility, similar to water or electricity. However, scheduling tasks across such a network is significantly more complex than in traditional data centers. CPNs are inherently dynamic, characterized by fluctuating network latency, heterogeneous nodes, and constantly changing resource availability.
Standard algorithms often fail in these environments. While Deep Reinforcement Learning (DRL) offers a potential solution, it frequently encounters the sparse reward problem. In a complex scheduling environment, an agent may not receive feedback on the quality of its decisions until many tasks are completed, leading to delayed signals. This makes it difficult for the agent to correlate specific actions with performance outcomes, resulting in slow training convergence and inefficient energy utilization.
Methods
To address these challenges, the C2RS (CPN Resource Scheduling with Reward Shaping) framework was developed. It bridges the gap between sparse feedback and optimal scheduling through several key components:
- Advantage Actor-Critic (A2C) Backbone: C2RS utilizes the A2C architecture to balance value estimation (assessing state quality) and policy gradients (determining the optimal action).
- Self-Supervised Reward Shaping: This core innovation involves a neural network that infers a dense reward function. By analyzing historical action-outcome trajectories and using a binary classification loss to rank them, C2RS generates a continuous feedback stream. This guides the agent at every step, rather than waiting for final task completion.
- Multi-Objective Optimization: The reward function incorporates multiple factors beyond simple speed, including job completion time vs. expectation, energy consumption, dependency delays (modeled as Directed Acyclic Graphs), and job valuation for prioritization.
Experimental Setup and Datasets
The framework was evaluated using a synthetic dataset modeled after the 2019 Google Cluster Trace. This ensured realistic job arrival rates and resource requirements. The dataset comprised over 10,000 jobs, with a mix of short-to-medium (80%) and long-duration (20%) tasks. System loads were varied from 10% to 190% to test the scheduler under extreme conditions.
Results and Discussion
C2RS was compared against four major DRL baselines: DeepRM, DeepJS, WSDRL, and Decima. The results demonstrated significant improvements across several metrics:
- Job Slowdown: C2RS achieved the lowest average job slowdown across all load levels, particularly outperforming DeepRM at high loads (150%+).
- Energy Efficiency: Under a 170% system load, C2RS reduced energy consumption by approximately 17% compared to DeepRM and 19% compared to DeepJS. It also showed a 10-12% improvement over WSDRL and Decima.
- Learning Efficiency: C2RS reached a stable, near-optimal policy in approximately 200 iterations, whereas competing methods remained significantly less efficient even after 1,000 iterations.
The "dense" reward signal allows the agent to learn the underlying network dynamics much faster than manual reward engineering. Furthermore, C2RS produced a more stable energy consumption curve by avoiding server contention and backlogs, leading to smoother operations.
Conclusion
C2RS demonstrates that self-supervised reward shaping can effectively overcome the sparse reward problem in decentralized Computing Power Networks. By providing continuous, dense feedback, the framework enables faster learning and more energy-efficient scheduling. Future work will focus on testing the framework in environments with extreme hardware heterogeneity and deploying it on live, global-scale networks to further validate its real-world performance.
Summary updated