Abstract
Station-keeping tasks for high-altitude balloons show promise in areas such as ecological surveys, atmospheric analysis, and communication relays. However, identifying the optimal time and position to launch a latex high-altitude balloon is still a challenging and multifaceted problem. We first illustrate how reinforcement learning-based controllers can exploit the environment, leading to risky behaviors like flying near the edge of a target region. We propose a Tanh-based reward modification that compensates for this exploitation, leading to higher performance on unseen data. Furthermore, we demonstrate how Bayesian Optimisation (BO) can efficiently identify optimal launch locations, often finding that the best sites are surprisingly far from the target region. Results show that BO identifies these locations in significantly fewer steps than traditional optimization methods.
Introduction
High-altitude balloons (HABs) are low-cost, long-endurance alternatives to satellites for atmospheric research and communication. However, as under-actuated systems, their horizontal movement is entirely dependent on wind currents. "Station-keeping"—maintaining a position within a specific geographic radius—requires navigating complex, non-linear wind patterns by only controlling the balloon's altitude. A critical challenge is determining not just how to pilot the balloon, but where and when to launch it to maximize its time within the target zone.
Methodology: A Two-Stage Optimization Framework
The researchers developed a framework that separates the problem into two distinct tasks: controller training and launch site optimization.
- Station-Keeping Controller: A Soft Actor-Critic (SAC) reinforcement learning algorithm is used to train the balloon's pilot. To prevent "reward hacking," where agents hover at the risky boundaries of a target zone, a Tanh-based reward function was introduced to incentivize staying near the center.
- Launch Location Optimisation: Bayesian Optimisation (BO) is used to search the high-dimensional space of launch coordinates and time offsets. BO uses a Gaussian Process surrogate model to intelligently select test points, maximizing the expected return of a mission rollout.
Addressing the "Edge Problem"
Standard reinforcement learning rewards often provide a flat value for being anywhere inside a target region. This can lead to agents "gaming" the system by staying at the perimeter, which offers little margin for error when wind patterns shift. The Tanh reward function provides a gradient that increases toward the center of the target region, forcing the agent to seek a safer, more robust position.
Key Results and Insights
- Improved Generalization: The Tanh reward function significantly outperformed the standard Step function on unseen test data (February 2023 wind patterns), averaging 23.20 steps in the target region versus 14.11 steps for the Step model. The success rate for reaching the target also increased from 29% to 49%.
- Counter-Intuitive Launch Sites: The study found that the optimal launch site is rarely within the target region itself. Because balloons require time to reach altitude and intercept favorable wind layers, launching from hundreds of kilometers away is often necessary to ensure the balloon arrives at the target at the ideal moment.
- Optimization Efficiency: Bayesian Optimization proved far more efficient than traditional methods like Particle Swarm Optimization (PSO). BO reached converged scores in roughly half the iterations required by PSO, demonstrating its utility for complex atmospheric mission planning.
Conclusion
By combining robust RL controllers with efficient Bayesian search methods, this framework provides a powerful tool for planning high-altitude balloon missions. The shift from simple boundary-based rewards to center-seeking incentives drastically improves real-world reliability, while BO offers a fast, automated way to solve the complex "where and when" of balloon launches.
Summary updated