Optimal Energy Allocation Policy for Wireless Networks in the Sky

Dinh Thai Hoang, Dusit Niyato, Nguyen Tai Hung

Abstract

Google's Project Loon was launched in 2013 with the aim of providing Internet access to rural and remote areas. In the Loon network, balloons travel around the Earth and bring access points to the users who cannot connect directly to the global wired Internet. The signals from the users will be transmitted through the balloon network to the base stations connected to the Internet service provider (ISP) on Earth. The process of transmitting and receiving data consume a certain amount of energy from the balloon, while the energy on balloons cannot be supplied by stable power source or by replacing batteries frequently. Instead, the balloons can harvest energy from natural energy sources, e.g., solar energy, or from radio frequency energy by equipping with appropriate circuits. However, such kinds of energy sources are often dynamic and thus how to use this energy efficiently is the main goal of this paper. In this paper, we study the optimal energy allocation problem for the balloons such that network performance is optimized and the revenue for service providers is maximized. We first formulate the stochastic optimization problem as a Markov decision process and then apply a learning algorithm based on simulation-based method to obtain optimal policies for the balloons. Numerical results obtained by extensive simulations clearly show the efficiency and convergence of the proposed learning algorithm.

Introduction

Providing internet access to remote or geographically challenging areas, such as oceans or mountainous regions, remains a significant hurdle for traditional ground-based infrastructure. Initiatives like Google’s Project Loon address this by deploying high-altitude balloons in the stratosphere to create a wireless mesh network. However, these balloons operate under severe energy constraints. They rely on stochastic energy harvesting from solar panels and radio frequency (RF) sources to power their access points.

Since battery capacity is limited and energy harvesting is unpredictable, balloons must intelligently decide which data requests to serve. A "greedy" policy—accepting every request as long as power is available—can lead to energy depletion, causing the system to miss high-priority, high-revenue requests later. This paper proposes an optimal admission control policy that balances energy conservation with profit maximization across various user classes, including ground users, other balloons, and satellites.

System Model and Methodology

The authors model the balloon's energy management as a stochastic optimization problem using a Markov Decision Process (MDP). The system state is defined by the current battery level and the type of incoming event, which could be a data request from one of three user classes or an energy arrival.

The proposed system model for admission control in a balloon-based wireless network.

The core of the methodology is an admission control policy where the system decides whether to accept or reject an incoming request. To avoid the "curse of dimensionality" associated with large state spaces, the authors employ a parameterized randomized policy. Instead of a rigid threshold, the system uses a probability-based function to make decisions. A simulation-based reinforcement learning algorithm (Policy Gradient) is then used to iteratively update the policy parameters, allowing the system to converge on the most profitable strategy without requiring perfect prior knowledge of the environment.

Performance Evaluation and Key Insights

The proposed approach was evaluated through extensive MATLAB simulations. Traffic and energy harvesting arrivals were modeled using Poisson distributions. Three classes of users were defined, with Class-1 (e.g., inter-balloon or satellite links) yielding higher rewards than Class-2 or Class-3 (e.g., standard ground users).

A key finding is that the learned policy naturally develops "soft thresholds." As shown in the figure below, the algorithm learns to prioritize high-reward requests. For instance, Class-1 requests are accepted with high probability even at low energy levels, whereas lower-reward classes are only accepted when the battery has a sufficient buffer.

Probability of accepting requests from different user classes based on the energy level in the queue.

Key Results:

  • Revenue Maximization: The learning algorithm achieved an average reward approximately 8.8% higher than the baseline greedy policy.
  • Energy Buffer: Unlike the greedy policy, the learned strategy maintains a higher average energy level, providing a safety buffer for critical operations.
  • Saturation Point: Increasing battery capacity only improves performance up to a certain point (15 units in this study), after which the system becomes limited by the energy harvesting rate rather than storage capacity.

Conclusion

This research demonstrates that intelligent energy allocation is crucial for the sustainability and profitability of "Internet in the sky" networks. By using a model-free learning approach, balloons can adapt to dynamic energy harvesting conditions and prioritize high-value traffic. Future work will explore multi-balloon coordination and more complex Quality of Service (QoS) requirements using multi-agent reinforcement learning to manage entire sky networks holistically.

Summary updated

Abstract

Google's Project Loon was launched in 2013 with the aim of providing Internet access to rural and remote areas. In the Loon network, balloons travel around the Earth and bring access points to the users who cannot connect directly to the global wired Internet. The signals from the users will be transmitted through the balloon network to the base stations connected to the Internet service provider (ISP) on Earth. The process of transmitting and receiving data consume a certain amount of energy from the balloon, while the energy on balloons cannot be supplied by stable power source or by replacing batteries frequently. Instead, the balloons can harvest energy from natural energy sources, e.g., solar energy, or from radio frequency energy by equipping with appropriate circuits. However, such kinds of energy sources are often dynamic and thus how to use this energy efficiently is the main goal of this paper. In this paper, we study the optimal energy allocation problem for the balloons such that network performance is optimized and the revenue for service providers is maximized. We first formulate the stochastic optimization problem as a Markov decision process and then apply a learning algorithm based on simulation-based method to obtain optimal policies for the balloons. Numerical results obtained by extensive simulations clearly show the efficiency and convergence of the proposed learning algorithm.
BibTeX
@inproceedings{hoang2015optimal,
  author = {Dinh Thai Hoang and Dusit Niyato and Nguyen Tai Hung},
  title = {{Optimal Energy Allocation Policy for Wireless Networks in the Sky}},
  booktitle = {ICC},
  year = {2015},
  doi = {10.1109/icc.2015.7248817},
}
Figure