Optimal energy allocation policy for wireless networks in the sky

Dinh Thai Hoang, Dusit Niyato, Nguyen Tai Hung

TL;DR

This paper proposes a smart energy management strategy for "Internet in the sky" balloon networks (like Google’s Project Loon). By modeling the balloon's energy state and data requests as a Markov Decision Process (MDP) and applying a simulation-based reinforcement learning algorithm, the authors demonstrate how balloons can maximize service provider revenue by selectively accepting data requests based on current energy levels and harvested power.

What Problem Does This Paper Address?

Providing internet to remote or geographically challenging areas (like oceans or mountains) is difficult with traditional ground-based infrastructure. Projects like Google Loon use high-altitude balloons to create a wireless mesh network in the stratosphere.

However, these balloons face a critical constraint: energy. They rely on stochastic (random) energy harvesting from solar panels and radio frequency (RF) sources to power their access points. Since batteries are limited in size and energy harvesting is unpredictable, the balloon must decide which data requests to serve. If it accepts every request (a "greedy" policy), it might run out of power and miss high-priority, high-revenue data transfers later. The challenge is to find an optimal "admission control" policy that balances energy conservation with profit maximization across different types of users (e.g., ground users, other balloons, or satellites).

Methods

The authors approach this as a stochastic optimization problem using the following framework:

  • Markov Decision Process (MDP): They define the system state by the current battery level and the type of incoming event (a request from one of three user classes or an energy arrival).
  • Admission Control Policy: The action at each step is binary: accept or reject the incoming data request.
  • Parameterized Randomized Policy: Instead of a hard threshold, they use a probability-based function (similar to a sigmoid) to decide on actions. This allows the system to "learn" the best parameters ($\Theta$) for the policy.
  • Policy Gradient & Simulation-Based Learning: To avoid the "curse of dimensionality" (where the problem becomes too complex to solve as states increase) and the need for perfect environmental knowledge, they use an online learning algorithm. This algorithm updates the policy parameters iteratively based on simulated interactions with the environment, eventually converging on the most profitable strategy.

Datasets

The paper does not use a pre-existing real-world dataset. Instead, the authors evaluate their approach using extensive MATLAB simulations.

  • Traffic Modeling: Request arrivals for three classes of users are modeled using Poisson distributions.
  • Energy Modeling: Energy harvesting arrivals are also modeled as a Poisson process.
  • Parameters: They set specific reward values for different classes (e.g., Class-1 requests from other balloons are worth more than Class-2 requests from ground users) to test the algorithm's ability to prioritize.

Key Insights and Ideas

  • Greedy is Suboptimal: The most intuitive policy—accepting every request if energy is available—is actually less efficient. It fails to "save" energy for high-value requests that might arrive when the battery is low.
  • Threshold Learning: The learned policy naturally develops "soft thresholds." For example, the algorithm learns to always accept high-reward Class-1 requests but only accepts lower-reward Class-2 requests if the battery level is above a certain point (e.g., 5 units).
  • The Energy Bottleneck: The authors found that increasing battery capacity only improves performance up to a certain point. Beyond a specific threshold (15 units in their simulation), the system performance saturates because it becomes limited by the rate of energy arrival rather than the storage capacity.
  • Model-Free Advantage: By using a simulation-based learning method, the system can adapt to environments where the exact probability of energy harvesting or request arrivals isn't known in advance.

Key Results

  • Revenue Increase: The proposed learning algorithm achieved an average reward approximately 8.8% higher than the greedy policy.
  • Convergence: The algorithm demonstrated stable convergence within $5 \cdot 10^5$ to $10^6$ iterations, proving it can effectively find the optimal policy in a complex state space.
  • Energy Efficiency: The learning algorithm maintained a higher average energy level in the battery compared to the greedy policy, providing a better safety buffer for the balloon's operations.

Limitations and Future Work

  • Simplified Traffic Models: The use of Poisson distributions for data and energy arrivals is a standard mathematical starting point but may not capture the bursty or seasonal nature of real-world internet traffic and solar energy.
  • Single-Node Focus: The paper focuses on the optimization of a single balloon's access point. In a real deployment, balloons move and hand off signals to one another.
  • Future Directions: The authors suggest that future research could involve multi-balloon coordination and more complex Quality of Service (QoS) requirements, potentially using multi-agent reinforcement learning to manage the entire "sky network" holistically.

Summary updated

Abstract

Google's Project Loon [1] was launched in 2013 with the aim of providing Internet access to rural and remote areas. In the Loon network, balloons travel around the Earth and bring access points to the users who cannot connect directly to the global wired Internet. The signals from the users will be transmitted through the balloon network to the base stations connected to the Internet service provider (ISP) on Earth. The process of transmitting and receiving data consume a certain amount of energy from the balloon, while the energy on balloons cannot be supplied by stable power source or by replacing batteries frequently. Instead, the balloons can harvest energy from natural energy sources, e.g., solar energy, or from radio frequency energy by equipping with appropriate circuits. However, such kinds of energy sources are often dynamic and thus how to use this energy efficiently is the main goal of this paper. In this paper, we study the optimal energy allocation problem for the balloons such that network performance is optimized and the revenue for service providers is maximized. We first formulate the stochastic optimization problem as a Markov decision process and then apply a learning algorithm based on simulation-based method to obtain optimal policies for the balloons. Numerical results obtained by extensive simulations clearly show the efficiency and convergence of the proposed learning algorithm.
BibTeX
@inproceedings{hoang2015optimal,
  author = {Dinh Thai Hoang and Dusit Niyato and Nguyen Tai Hung},
  title = {{Optimal energy allocation policy for wireless networks in the sky}},
  booktitle = {ICC},
  year = {2015},
  doi = {10.1109/icc.2015.7248817},
}

Loading related content...

Figure