Abstract
This paper focuses on finding reinforcement learning policies for control systems with hard state and action constraints. Despite its success in many domains, reinforcement learning is challenging to apply to problems with hard constraints, especially if both the state variables and actions are constrained. Previous works seeking to ensure constraint satisfaction, or safety, have focused on adding a projection step to a learned policy. Yet, this approach requires solving an optimization problem at every policy execution step, which can lead to significant computational costs. To tackle this problem, this paper proposes a new approach, termed Vertex Networks (VNs), with guarantees on safety during exploration and on learned control policies by incorporating the safety constraints into the policy network architecture. Leveraging the geometric property that all points within a convex set can be represented as the convex combination of its vertices, the proposed algorithm first learns the convex combination weights and then uses these weights along with the pre-calculated vertices to output an action. The output action is guaranteed to be safe by construction. Numerical examples illustrate that the proposed VN algorithm outperforms vanilla reinforcement learning in a variety of benchmark control tasks.
Introduction
Reinforcement Learning (RL) has demonstrated remarkable potential in virtual environments, yet its application to real-world physical systems—such as robotics and autonomous driving—remains constrained by safety concerns. Real-world systems are governed by hard constraints: a robotic manipulator must avoid collisions, and autonomous vehicles must adhere to speed and stability limits.
Traditional approaches to "Safe RL" typically rely on soft penalties or projection methods. Soft penalties discourage unsafe behavior through negative rewards but offer no formal guarantees, often leading to crashes during the exploration phase. Projection methods ensure safety by mapping unsafe actions back into a feasible set, but they require solving computationally intensive optimization problems (e.g., Quadratic Programs) at every time step, hindering real-time performance. This paper introduces Vertex Networks (VNs), a novel architecture that ensures safety by construction while remaining computationally efficient for real-time control.
Methods: The Vertex Network Architecture
The proposed method targets control-affine systems, where the system dynamics are linear with respect to the control input. The core innovation of the VN is the integration of a "Safe Layer" directly into the neural network architecture.
Geometric Foundation and Dynamic Safety Sets
The approach leverages a fundamental property of convex geometry: any point within a convex polytope can be uniquely represented as a convex combination of its vertices. At each time step $t$, the algorithm identifies a "Safe Action Set" $\mathcal{U}_t$. This set is defined by the intersection of physical actuator limits and the set of actions that maintain the system within safe state boundaries.
The Safe Layer
Instead of directly outputting an action, the feed-forward network outputs a vector of weights $\lambda$. These weights are passed through a softmax layer to ensure they are non-negative and sum to one. These normalized weights $\bar{\lambda}_i$ are then used to compute a convex combination of the pre-calculated vertices $P_i^{(t)}$ of the safe action set.
By construction, the resulting action $u_t = \sum_i \bar{\lambda}_i P_i^{(t)}$ is guaranteed to lie within the safe polytope. Since the entire operation is differentiable, the network can be trained end-to-end using standard model-free RL algorithms like Deep Deterministic Policy Gradient (DDPG).
Experimental Evaluation
The VN approach was evaluated on three benchmark control tasks modified with hard constraints: an Inverted Pendulum with torque and angle limits, a Mass-Spring system with velocity constraints, and a Hovercraft tracking task with strict tilt angle limits.
Key Results
The experimental results highlight several advantages of VNs over standard Policy Networks (PN) with soft penalties:
- Guaranteed Safety: In all benchmarks, the VN maintained zero constraint violations throughout the entire training process. As shown in the performance metrics, while the baseline PN frequently violated constraints (especially early in training), the VN remained consistently within the safe region.
- Improved Learning Efficiency: VNs often achieved higher rewards faster than the baseline. The safe layer effectively prunes the search space, preventing the agent from exploring catastrophic actions and guiding it toward optimal, safe behavior.
- Handling Tight Constraints: In the Hovercraft task, the VN successfully respected a very narrow tilt constraint that the baseline PN failed to satisfy even after extensive training.
Conclusion
Vertex Networks provide a robust framework for safe reinforcement learning by embedding safety constraints into the network's output layer. By utilizing the geometric properties of convex polytopes, VNs ensure safety by construction during both training and deployment. While the current formulation requires known system dynamics and faces scaling challenges in very high-dimensional action spaces, it represents a significant step toward deploying RL in safety-critical real-world applications.
Summary updated