Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing

Jie Xu, Lixing Chen, Shaolei Ren

Abstract

Mobile Edge Computing (MEC) brings cloud-like processing closer to users, but powering these remote servers with unpredictable renewable energy is a major challenge. This paper introduces a novel reinforcement learning algorithm based on "Post-Decision States" (PDS) to dynamically manage workload offloading and server autoscaling. The approach learns optimal, foresighted policies significantly faster than standard Q-learning, minimizing both service delays and the need for expensive backup power.

Introduction: The Challenge of Green Edge Computing

Mobile Edge Computing (or Fog Computing) reduces latency by processing data at the network edge—such as cellular base stations—rather than sending it to a distant centralized cloud. However, providing a reliable grid power supply to remote or rugged edge locations is often prohibitively expensive or ecologically damaging.

Using on-site renewable energy (solar or wind) is a sustainable alternative, but it introduces significant variability. If the sun isn't shining or the wind isn't blowing, edge servers may deplete their batteries. To maintain service quality, the system must constantly decide:

  1. Offloading: How much workload should be processed locally versus offloaded to the cloud?
  2. Autoscaling: How many local edge servers should be powered on (active) versus kept inactive?
System architecture for energy-harvesting MEC, showing the interplay between workload, edge devices, renewable energy sources, and the power grid.

Making these decisions is difficult because future workload spikes, network congestion, and weather changes are unknown. Aggressive local processing risks draining the battery and forcing the use of expensive backup generators, while conservative strategies lead to high network latency.

Methodology: Post-Decision State Learning

The authors formulate this problem as a Markov Decision Process (MDP) to minimize long-term system costs, including delay, battery depreciation, and backup power usage. Because environmental dynamics are unknown, Reinforcement Learning (RL) is employed to learn the optimal policy online.

Standard RL algorithms like Q-learning often suffer from the "curse of dimensionality," requiring extensive exploration to converge in large state spaces. To overcome this, the authors introduce a Post-Decision State (PDS) based learning algorithm.

Table of symbols and definitions used in the MDP formulation and PDS algorithm.

The PDS approach looks at the intermediate virtual state after a decision is made (e.g., power allocation) but before random variables (e.g., harvested energy) are realized. This separates the deterministic components—like battery drainage—from stochastic components like weather and traffic. This separation allows for "batch updates" across multiple states, drastically accelerating the learning process.

Furthermore, the authors mathematically prove that the optimal computing power demand follows a monotone policy structure, where demand is non-decreasing relative to available battery power. This theoretical insight simplifies practical implementation.

Experimental Evaluation

The algorithm was evaluated in a simulated environment with a 15-minute time slot resolution over 10,000 slots. The simulation modeled:

  • Workload: Arrival rates between 10–100 units/sec.
  • Network: Congestion states ranging from 20–60 ms/unit.
  • Environment: Three states (Low, Medium, High) governing green energy harvesting.
  • Hardware: A 2 kWh battery capacity.

Key Results

  • Faster Convergence: The PDS-based algorithm reached an optimal policy significantly faster than standard Q-learning, maintaining a lower time-average cost throughout the simulation.
  • Reduced Backup Costs: By intelligently managing battery levels, the PDS algorithm slashed reliance on backup diesel generators. Backup power costs were only 12.6% of the total cost, compared to 46.9% for myopic (short-sighted) optimization.
  • Optimized Energy Harvesting: Unlike fixed-power strategies, the PDS algorithm keeps the battery at a safe yet low state, ensuring there is always capacity to capture newly harvested green energy and minimizing waste.

Conclusion and Future Directions

The PDS-based learning framework effectively balances the trade-offs between latency and energy sustainability in MEC. While the current model focuses on a single edge node with ideal battery assumptions, future work aims to scale this to multi-node systems. A promising direction is "green power-aware geographical load balancing," which would route workloads across a network of edge nodes based on localized renewable energy availability.

Summary updated

Abstract

Mobile edge computing (also known as fog computing) has recently emerged to enable <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-situ</i> processing of delay-sensitive applications at the edge of mobile networks. Providing grid power supply in support of mobile edge computing, however, is costly and even infeasible (in certain rugged or under-developed areas), thus mandating on-site renewable energy as a major or even sole power supply in increasingly many scenarios. Nonetheless, the high intermittency and unpredictability of renewable energy make it very challenging to deliver a high quality of service to users in energy harvesting mobile edge computing systems. In this paper, we address the challenge of incorporating renewables into mobile edge computing and propose an efficient reinforcement learning-based resource management algorithm, which learns on-the-fly the optimal policy of dynamic workload offloading (to the centralized cloud) and edge server provisioning to minimize the long-term system cost (including both service delay and operational cost). Our online learning algorithm uses a decomposition of the (offline) value iteration and (online) reinforcement learning, thus achieving a significant improvement of learning rate and run-time performance when compared to standard reinforcement learning algorithms such as <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -learning. We prove the convergence of the proposed algorithm and analytically show that the learned policy has a simple monotone structure amenable to practical implementation. Our simulation results validate the efficacy of our algorithm, which significantly improves the edge computing performance compared to fixed or myopic optimization schemes and conventional reinforcement learning algorithms.
BibTeX
@article{xu2017online,
  author = {Jie Xu and Lixing Chen and Shaolei Ren},
  title = {{Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing}},
  journal = {IEEE Trans. Cogn. Commun. Netw.},
  year = {2017},
  url = {https://openreview.net/forum?id=m0BcIvNvMi},
}
Figure