Abstract
Safety remains a critical barrier to the deployment of deep reinforcement learning (RL) in real-world control tasks. While constrained reinforcement learning (CRL) frameworks utilize cost functions to manage safety within Constrained Markov Decision Processes (CMDPs), they frequently fail to achieve zero-violation performance—even when the cost limit is set to zero. This paper investigates the root causes of this failure, identifying the lack of informative cost signals as a primary culprit. We introduce AutoCost, a framework that employs evolutionary algorithms to automatically search for "intrinsic" cost functions that enable zero-violation performance. Evaluated on the Safety Gym benchmark, our results demonstrate that agents augmented with these evolved cost functions achieve zero constraint violations and maintain performance comparable to baseline agents across various tasks and robot morphologies.
The Challenge of Zero-Violation Safety
In standard Safe RL, agents are typically trained to maximize rewards while keeping an extrinsic cost ($c_{ex}$) below a specific threshold. However, state-of-the-art algorithms like Constrained Policy Optimization (CPO) and PPO-Lagrangian often suffer from "residual" violations. Even when the safety threshold is zero, these agents continue to collide with obstacles or enter hazardous zones.
This failure stems from the nature of the safety signals themselves. Most environments provide "indicator functions"—binary signals that only trigger after a violation has occurred. This sparsity makes it difficult for an agent to learn the boundaries of safe behavior until it is too late. While human experts can manually design "dense" cost functions (e.g., penalizing an agent based on its proximity to a hazard), this process is labor-intensive, task-specific, and requires significant trial-and-error tuning.
The AutoCost Framework
AutoCost addresses this by automating the design of dense cost functions through a bi-level optimization process. The framework introduces an intrinsic cost ($c_{in}$), represented by a small Multi-Layer Perceptron (MLP) that processes raw sensor data, such as LiDAR.
1. The Inner Loop: Agent Training
In the inner loop, a standard Safe RL agent is trained in the environment. However, instead of relying solely on the sparse extrinsic cost provided by the environment, the agent's total cost signal is the sum of the extrinsic and the evolved intrinsic costs ($c = c_{ex} + c_{in}$).
2. The Outer Loop: Evolutionary Search
AutoCost uses an Evolutionary Strategy (ES) to optimize the parameters of the intrinsic cost MLP. The process follows these steps:
- Population Initialization: A population of 50 candidate cost functions is generated.
- Evaluation: Each candidate is used to train an RL agent. The "fitness" of the cost function is determined first by the agent's ability to achieve zero violations, and secondarily by the total reward accumulated.
- Selection and Mutation: The highest-performing cost functions are selected to seed the next generation. Mutations are applied via Gaussian noise and scaling to explore new potential cost structures.
Through this process, the framework discovers a robust cost function termed IC4SRL (Intrinsic Cost for Safe RL).
Experimental Results and Insights
We evaluated AutoCost using the Safety Gym benchmark, testing across multiple tasks ("Goal" and "Push") and robot models, including the simple Point robot, the wheeled Car, and the complex quadruped Doggo.
Achieving Zero Violations
In the Goal-Hazard-Point environment, agents using the IC4SRL-augmented cost signal achieved zero constraint violations upon convergence. In contrast, baseline agents using only extrinsic costs continued to experience collisions throughout training. This suggests that the evolved cost function provides the proactive signal necessary for true safety.
The "Buffer Zone" Effect
Analysis of the evolved IC4SRL function reveals that it naturally creates a "safety margin." Rather than only firing at the moment of impact, the function assigns increasing costs to states as the agent approaches a hazard. This creates a potential field that effectively pushes the agent away from danger before a violation occurs, compensating for the lag in standard RL optimization.
Generalization and Robustness
One of the most significant findings was the transferability of the evolved cost function. A cost function evolved using a simple Point robot was successfully applied to the Car robot and the Push task. In these new scenarios, the agents maintained zero-violation performance without any additional tuning of the cost parameters. Furthermore, the search process identified cost functions that were robust across different RL algorithms (e.g., both CPO and PPO-Lagrangian), overcoming the sensitivity issues often found in hand-tuned rewards.
Limitations and Future Directions
While AutoCost represents a significant step toward autonomous safety design, several challenges remain:
- Complexity Scaling: The evolved cost function failed to generalize to the "Doggo" robot. This was primarily because the underlying RL agent struggled to learn any stable locomotion or task behavior on the complex quadruped, rendering the safety signal ineffective.
- Sensor Dependency: The current IC4SRL is coupled to LiDAR sensor inputs. Future research will focus on developing more universal state representations to allow cost functions to transfer across different sensor suites.
- Computational Intensity: The evolutionary search is computationally demanding, requiring approximately 54 hours in the tested setup. However, the authors note that this is a one-time "offline" cost; once discovered, the intrinsic cost function can be deployed instantly to train new agents.
Conclusion
AutoCost demonstrates that the primary bottleneck in Safe RL is often the quality of the cost signal rather than the optimization algorithm itself. By evolving dense, proactive intrinsic costs, we can bridge the gap between "constrained" RL and truly "zero-violation" RL, moving closer to the safe deployment of autonomous systems in the real world.
Summary updated