Reinforcement Learning for Docking Maneuvers with Prescribed Performance

Simon Gottschalk, Lukas Lanza, Karl Worthmann, Kerstin Lux-Gottschalk

TL;DR

This paper introduces a hybrid control strategy for satellite docking that combines the flexibility of Reinforcement Learning (RL) with the rigorous safety guarantees of "Funnel Control." By using Funnel Control as a high-gain safety filter, the authors ensure that a satellite stays within predefined performance bounds even while the RL agent is still learning, effectively preventing collisions during the training phase.

What Problem Does This Paper Address?

Satellite docking is a high-stakes maneuver. Traditionally, this is handled by solving complex optimal control problems (like Model Predictive Control). However, as space becomes more crowded with debris and new satellites, the number of variables and potential collision scenarios has exploded, pushing classical methods to their computational limits.

Reinforcement Learning (RL) offers a promising alternative because it can handle high-dimensional systems and adapt to various scenarios without recomputing solutions from scratch. The "catch" is that RL is notoriously difficult to provide safety guarantees for—especially during the "exploration" phase of learning where the agent often tries erratic actions. In space, a single "erratic action" during training could mean a catastrophic collision. This paper seeks to bridge the gap between the optimality of RL and the safety requirements of aerospace engineering.

Methods

The authors propose a two-component controller:

  1. The RL Component: They use the Proximal Policy Optimization (PPO) algorithm, a popular gradient-based RL method, to learn an optimal control policy. The agent's goal is to move a 9-degree-of-freedom satellite (body + robotic arm) to a target position.
  2. The Funnel Control Component: This acts as a "safeguard." Funnel control is a model-free, high-gain adaptive feedback controller. It doesn't need to know the exact equations of the system; it simply monitors the error (the distance between the current state and a safe reference trajectory). If the error gets too close to a predefined "funnel" boundary, the controller ramps up its gain to "push" the system back into the safe zone.
  3. The Integration (Dwell-Time Activation): To prevent the two controllers from fighting each other or causing "chattering" (rapid, jerky switching), the authors implemented a dwell-time activation scheme. The funnel controller only kicks in when a safety threshold is breached and stays active for a set minimum time to ensure stability.

Datasets

The study does not use a static dataset. Instead, it utilizes a custom-built physics simulation of a satellite based on the multibody system modeling of Kortüm and Lugner (1993).

  • State Space: 18 variables (positions and velocities of the satellite body and the 2-link robotic arm).
  • Action Space: 9 independent control inputs (forces for the body and torques for the arm joints).
  • Environment: Includes simulated "collision areas" (like solar panels) where rewards become highly negative.

Key Insights and Ideas

  • Safety Without a Model: One of the most powerful aspects of Funnel Control is that it provides tracking guarantees for unknown nonlinear systems. This makes it a perfect "safety filter" for RL, which is often used precisely because the environment is complex or hard to model perfectly.
  • Reward Shaping for Safety: The authors didn't just use the funnel to stop crashes; they included the "funnel intervention" in the RL reward function. By penalizing the RL agent whenever the safety filter had to step in, the agent eventually learned to mimic the safe behavior of the funnel controller.
  • The "Aha" Moment on Chattering: Simply switching between RL and Funnel Control based on a threshold leads to instability. The inclusion of a "dwell-time"—a memory of recent safety breaches—is what makes the combined controller smooth enough for real-world application.

Key Results

  • Zero Violations: In simulations of 100 trajectories, the hybrid "Funnel-RL" controller resulted in zero safety violations, even at the very beginning of the training process when the RL policy was essentially random.
  • Pure RL Failure: In contrast, a standard RL agent (without the funnel) had a high number of violations early on. While it eventually learned to be safe after about 350 iterations, it offered no guarantees during that learning period.
  • Convergence: Over time, the RL agent learned to operate entirely within the safe "funnel." By the end of training, the funnel controller rarely had to intervene, showing that the RL agent had successfully learned a high-performance and safe policy.

Limitations and Future Work

The authors acknowledge that while Funnel Control is "model-free," it still relies on certain structural assumptions about the system (like having a well-defined "relative degree" and stable internal dynamics).

Future directions include: * Refining the design of the safe/safety-critical regions to better account for uncertainties in model parameters. * Investigating how "trustworthy" the training state is to determine when the safety filter can be dialed back. * Testing the approach on even more complex docking scenarios with tumbling targets.

Summary updated

Abstract

We propose a two-component data-driven controller to safely perform docking maneuvers for satellites. Reinforcement Learning is used to deduce an optimal control policy based on measurement data. To safeguard the learning phase, an additional feedback law is implemented in the control unit, which guarantees the evolution of the system within predefined performance bounds. We define safe and safety-critical areas to train the feedback controller based on actual measurements. To avoid chattering, a dwell-time activation scheme is implemented. We provide numerical evidence for the performance of the proposed controller for a satellite docking maneuver with collision avoidance.
BibTeX
@inproceedings{gottschalk2024reinforcement,
  author = {Simon Gottschalk and Lukas Lanza and Karl Worthmann and Kerstin Lux-Gottschalk},
  title = {{Reinforcement Learning for Docking Maneuvers with Prescribed Performance}},
  booktitle = {IFAC-PapersOnLine},
  year = {2024},
  doi = {10.1016/j.ifacol.2024.10.163},
}
Figure