Abstract
Mobile Edge Computing (MEC) brings cloud-like processing closer to users, but powering these remote servers with unpredictable renewable energy is a major challenge. This paper introduces a novel reinforcement learning algorithm based on "Post-Decision States" (PDS) to dynamically manage workload offloading and server autoscaling. The approach learns optimal, foresighted policies significantly faster than standard Q-learning, minimizing both service delays and the need for expensive backup power.
Introduction: The Challenge of Green Edge Computing
Mobile Edge Computing (or Fog Computing) reduces latency by processing data at the network edge—such as cellular base stations—rather than sending it to a distant centralized cloud. However, providing a reliable grid power supply to remote or rugged edge locations is often prohibitively expensive or ecologically damaging.
Using on-site renewable energy (solar or wind) is a sustainable alternative, but it introduces significant variability. If the sun isn't shining or the wind isn't blowing, edge servers may deplete their batteries. To maintain service quality, the system must constantly decide:
- Offloading: How much workload should be processed locally versus offloaded to the cloud?
- Autoscaling: How many local edge servers should be powered on (active) versus kept inactive?
Making these decisions is difficult because future workload spikes, network congestion, and weather changes are unknown. Aggressive local processing risks draining the battery and forcing the use of expensive backup generators, while conservative strategies lead to high network latency.
Methodology: Post-Decision State Learning
The authors formulate this problem as a Markov Decision Process (MDP) to minimize long-term system costs, including delay, battery depreciation, and backup power usage. Because environmental dynamics are unknown, Reinforcement Learning (RL) is employed to learn the optimal policy online.
Standard RL algorithms like Q-learning often suffer from the "curse of dimensionality," requiring extensive exploration to converge in large state spaces. To overcome this, the authors introduce a Post-Decision State (PDS) based learning algorithm.
The PDS approach looks at the intermediate virtual state after a decision is made (e.g., power allocation) but before random variables (e.g., harvested energy) are realized. This separates the deterministic components—like battery drainage—from stochastic components like weather and traffic. This separation allows for "batch updates" across multiple states, drastically accelerating the learning process.
Furthermore, the authors mathematically prove that the optimal computing power demand follows a monotone policy structure, where demand is non-decreasing relative to available battery power. This theoretical insight simplifies practical implementation.
Experimental Evaluation
The algorithm was evaluated in a simulated environment with a 15-minute time slot resolution over 10,000 slots. The simulation modeled:
- Workload: Arrival rates between 10–100 units/sec.
- Network: Congestion states ranging from 20–60 ms/unit.
- Environment: Three states (Low, Medium, High) governing green energy harvesting.
- Hardware: A 2 kWh battery capacity.
Key Results
- Faster Convergence: The PDS-based algorithm reached an optimal policy significantly faster than standard Q-learning, maintaining a lower time-average cost throughout the simulation.
- Reduced Backup Costs: By intelligently managing battery levels, the PDS algorithm slashed reliance on backup diesel generators. Backup power costs were only 12.6% of the total cost, compared to 46.9% for myopic (short-sighted) optimization.
- Optimized Energy Harvesting: Unlike fixed-power strategies, the PDS algorithm keeps the battery at a safe yet low state, ensuring there is always capacity to capture newly harvested green energy and minimizing waste.
Conclusion and Future Directions
The PDS-based learning framework effectively balances the trade-offs between latency and energy sustainability in MEC. While the current model focuses on a single edge node with ideal battery assumptions, future work aims to scale this to multi-node systems. A promising direction is "green power-aware geographical load balancing," which would route workloads across a network of edge nodes based on localized renewable energy availability.
Summary updated