AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning

Tairan He, Weiye Zhao, Changliu Liu

Abstract

Safety remains a critical barrier to the deployment of deep reinforcement learning (RL) in real-world control tasks. While constrained reinforcement learning (CRL) frameworks utilize cost functions to manage safety within Constrained Markov Decision Processes (CMDPs), they frequently fail to achieve zero-violation performance—even when the cost limit is set to zero. This paper investigates the root causes of this failure, identifying the lack of informative cost signals as a primary culprit. We introduce AutoCost, a framework that employs evolutionary algorithms to automatically search for "intrinsic" cost functions that enable zero-violation performance. Evaluated on the Safety Gym benchmark, our results demonstrate that agents augmented with these evolved cost functions achieve zero constraint violations and maintain performance comparable to baseline agents across various tasks and robot morphologies.

The Challenge of Zero-Violation Safety

In standard Safe RL, agents are typically trained to maximize rewards while keeping an extrinsic cost ($c_{ex}$) below a specific threshold. However, state-of-the-art algorithms like Constrained Policy Optimization (CPO) and PPO-Lagrangian often suffer from "residual" violations. Even when the safety threshold is zero, these agents continue to collide with obstacles or enter hazardous zones.

This failure stems from the nature of the safety signals themselves. Most environments provide "indicator functions"—binary signals that only trigger after a violation has occurred. This sparsity makes it difficult for an agent to learn the boundaries of safe behavior until it is too late. While human experts can manually design "dense" cost functions (e.g., penalizing an agent based on its proximity to a hazard), this process is labor-intensive, task-specific, and requires significant trial-and-error tuning.

The AutoCost Framework

AutoCost addresses this by automating the design of dense cost functions through a bi-level optimization process. The framework introduces an intrinsic cost ($c_{in}$), represented by a small Multi-Layer Perceptron (MLP) that processes raw sensor data, such as LiDAR.

1. The Inner Loop: Agent Training

In the inner loop, a standard Safe RL agent is trained in the environment. However, instead of relying solely on the sparse extrinsic cost provided by the environment, the agent's total cost signal is the sum of the extrinsic and the evolved intrinsic costs ($c = c_{ex} + c_{in}$).

AutoCost uses an Evolutionary Strategy (ES) to optimize the parameters of the intrinsic cost MLP. The process follows these steps:

  • Population Initialization: A population of 50 candidate cost functions is generated.
  • Evaluation: Each candidate is used to train an RL agent. The "fitness" of the cost function is determined first by the agent's ability to achieve zero violations, and secondarily by the total reward accumulated.
  • Selection and Mutation: The highest-performing cost functions are selected to seed the next generation. Mutations are applied via Gaussian noise and scaling to explore new potential cost structures.

Through this process, the framework discovers a robust cost function termed IC4SRL (Intrinsic Cost for Safe RL).

Experimental Results and Insights

We evaluated AutoCost using the Safety Gym benchmark, testing across multiple tasks ("Goal" and "Push") and robot models, including the simple Point robot, the wheeled Car, and the complex quadruped Doggo.

Achieving Zero Violations

In the Goal-Hazard-Point environment, agents using the IC4SRL-augmented cost signal achieved zero constraint violations upon convergence. In contrast, baseline agents using only extrinsic costs continued to experience collisions throughout training. This suggests that the evolved cost function provides the proactive signal necessary for true safety.

The "Buffer Zone" Effect

Analysis of the evolved IC4SRL function reveals that it naturally creates a "safety margin." Rather than only firing at the moment of impact, the function assigns increasing costs to states as the agent approaches a hazard. This creates a potential field that effectively pushes the agent away from danger before a violation occurs, compensating for the lag in standard RL optimization.

Generalization and Robustness

One of the most significant findings was the transferability of the evolved cost function. A cost function evolved using a simple Point robot was successfully applied to the Car robot and the Push task. In these new scenarios, the agents maintained zero-violation performance without any additional tuning of the cost parameters. Furthermore, the search process identified cost functions that were robust across different RL algorithms (e.g., both CPO and PPO-Lagrangian), overcoming the sensitivity issues often found in hand-tuned rewards.

Limitations and Future Directions

While AutoCost represents a significant step toward autonomous safety design, several challenges remain:

  • Complexity Scaling: The evolved cost function failed to generalize to the "Doggo" robot. This was primarily because the underlying RL agent struggled to learn any stable locomotion or task behavior on the complex quadruped, rendering the safety signal ineffective.
  • Sensor Dependency: The current IC4SRL is coupled to LiDAR sensor inputs. Future research will focus on developing more universal state representations to allow cost functions to transfer across different sensor suites.
  • Computational Intensity: The evolutionary search is computationally demanding, requiring approximately 54 hours in the tested setup. However, the authors note that this is a one-time "offline" cost; once discovered, the intrinsic cost function can be deployed instantly to train new agents.

Conclusion

AutoCost demonstrates that the primary bottleneck in Safe RL is often the quality of the cost signal rather than the optimization algorithm itself. By evolving dense, proactive intrinsic costs, we can bridge the gap between "constrained" RL and truly "zero-violation" RL, moving closer to the safe deployment of autonomous systems in the real world.

Summary updated

Abstract

Safety is a critical hurdle that limits the application of deep reinforcement learning (RL) to real-world control tasks. To this end, constrained reinforcement learning leverages cost functions to improve safety in constrained Markov decision processes. However, such constrained RL methods fail to achieve zero violation even when the cost limit is zero. This paper analyzes the reason for such failure, which suggests that a proper cost function plays an important role in constrained RL. Inspired by the analysis, we propose AutoCost, a simple yet effective framework that automatically searches for cost functions that help constrained RL to achieve zero-violation performance. We validate the proposed method and the searched cost function on the safe RL benchmark Safety Gym. We compare the performance of augmented agents that use our cost function to provide additive intrinsic costs with baseline agents that use the same policy learners but with only extrinsic costs. Results show that the converged policies with intrinsic costs in all environments achieve zero constraint violation and comparable performance with baselines.
BibTeX
@inproceedings{he2023autocost,
  author = {Tairan He and Weiye Zhao and Changliu Liu},
  title = {{AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26734},
}
Figure