Identifying Optimal Launch Sites of High-Altitude Latex-Balloons using Bayesian Optimisation for the Task of Station-Keeping

Jack Saunders, Sajad Saeedi, Adam Hartshorne, Binbin Xu, Özgür Şimşek, Alan J. Hunter, Wenbin Li

Abstract

Station-keeping tasks for high-altitude balloons show promise in areas such as ecological surveys, atmospheric analysis, and communication relays. However, identifying the optimal time and position to launch a latex high-altitude balloon is still a challenging and multifaceted problem. We first illustrate how reinforcement learning-based controllers can exploit the environment, leading to risky behaviors like flying near the edge of a target region. We propose a Tanh-based reward modification that compensates for this exploitation, leading to higher performance on unseen data. Furthermore, we demonstrate how Bayesian Optimisation (BO) can efficiently identify optimal launch locations, often finding that the best sites are surprisingly far from the target region. Results show that BO identifies these locations in significantly fewer steps than traditional optimization methods.

Introduction

High-altitude balloons (HABs) are low-cost, long-endurance alternatives to satellites for atmospheric research and communication. However, as under-actuated systems, their horizontal movement is entirely dependent on wind currents. "Station-keeping"—maintaining a position within a specific geographic radius—requires navigating complex, non-linear wind patterns by only controlling the balloon's altitude. A critical challenge is determining not just how to pilot the balloon, but where and when to launch it to maximize its time within the target zone.

Methodology: A Two-Stage Optimization Framework

The researchers developed a framework that separates the problem into two distinct tasks: controller training and launch site optimization.

  1. Station-Keeping Controller: A Soft Actor-Critic (SAC) reinforcement learning algorithm is used to train the balloon's pilot. To prevent "reward hacking," where agents hover at the risky boundaries of a target zone, a Tanh-based reward function was introduced to incentivize staying near the center.
  2. Launch Location Optimisation: Bayesian Optimisation (BO) is used to search the high-dimensional space of launch coordinates and time offsets. BO uses a Gaussian Process surrogate model to intelligently select test points, maximizing the expected return of a mission rollout.
The proposed two-stage framework: Launch Location Optimization using Bayesian Optimization and the Station-Keeping Task using Reinforcement Learning.

Addressing the "Edge Problem"

Standard reinforcement learning rewards often provide a flat value for being anywhere inside a target region. This can lead to agents "gaming" the system by staying at the perimeter, which offers little margin for error when wind patterns shift. The Tanh reward function provides a gradient that increases toward the center of the target region, forcing the agent to seek a safer, more robust position.

Comparison of balloon trajectories under Step (left) and Tanh (right) reward functions. The Tanh function encourages center-seeking behavior, avoiding the risky boundaries of the target region and improving generalization to unseen weather.

Key Results and Insights

  • Improved Generalization: The Tanh reward function significantly outperformed the standard Step function on unseen test data (February 2023 wind patterns), averaging 23.20 steps in the target region versus 14.11 steps for the Step model. The success rate for reaching the target also increased from 29% to 49%.
  • Counter-Intuitive Launch Sites: The study found that the optimal launch site is rarely within the target region itself. Because balloons require time to reach altitude and intercept favorable wind layers, launching from hundreds of kilometers away is often necessary to ensure the balloon arrives at the target at the ideal moment.
  • Optimization Efficiency: Bayesian Optimization proved far more efficient than traditional methods like Particle Swarm Optimization (PSO). BO reached converged scores in roughly half the iterations required by PSO, demonstrating its utility for complex atmospheric mission planning.

Conclusion

By combining robust RL controllers with efficient Bayesian search methods, this framework provides a powerful tool for planning high-altitude balloon missions. The shift from simple boundary-based rewards to center-seeking incentives drastically improves real-world reliability, while BO offers a fast, automated way to solve the complex "where and when" of balloon launches.

Summary updated

Abstract

Station-keeping tasks for high-altitude balloons show promise in areas such as ecological surveys, atmospheric analysis, and communication relays. However, identifying the optimal time and position to launch a latex high-altitude balloon is still a challenging and multifaceted problem. For example, tasks such as forest fire tracking place geometric constraints on the launch location of the balloon. Furthermore, identifying the most optimal location also heavily depends on atmospheric conditions. We first illustrate how reinforcement learning-based controllers, frequently used for station-keeping tasks, can exploit the environment. This exploitation can degrade performance on unseen weather patterns and affect station-keeping performance when identifying an optimal launch configuration. Valuing all states equally in the region, the agent exploits the region's geometry by flying near the edge, leading to risky behaviours. We propose a modification which compensates for this exploitation and finds this leads to, on average, higher steps within the target region on unseen data. Then, we illustrate how Bayesian Optimisation (BO) can identify the optimal launch location to perform station-keeping tasks, maximising the expected undiscounted return from a given rollout. We show BO can find this launch location in fewer steps compared to other optimisation methods. Results indicate that, surprisingly, the most optimal location to launch from is not commonly within the target region. Please find further information about our project at https://sites.google.com/view/bo-lauch-balloon/.
BibTeX
@misc{saunders2024identifying,
  author = {Jack Saunders and Sajad Saeedi and Adam Hartshorne and Binbin Xu and Özgür Şimşek and Alan J. Hunter and Wenbin Li},
  title = {{Identifying Optimal Launch Sites of High-Altitude Latex-Balloons using Bayesian Optimisation for the Task of Station-Keeping}},
  year = {2024},
  eprint = {2403.10784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10784},
}
Figure