AutoCost: Evolving Intrinsic Cost for Zero-Violation Reinforcement Learning

Tairan He, Weiye Zhao, Changliu Liu

TL;DR

AutoCost is a novel framework that uses evolutionary algorithms to automatically design "intrinsic" cost functions for Safe Reinforcement Learning. By searching for cost functions that provide dense, proactive safety signals, it enables RL agents to achieve true zero-violation performance in complex environments where standard methods typically fail.

What Problem Does This Paper Address?

In the world of Reinforcement Learning (RL), "safety" is usually handled via Constrained Markov Decision Processes (CMDPs). In this setup, an agent tries to maximize rewards while keeping a "cost" (representing safety violations) below a certain threshold.

The problem? Even when researchers set that threshold to zero, state-of-the-art Safe RL algorithms like CPO and PPO-Lagrangian almost always suffer from "residual" violations. They might reduce crashes, but they don't eliminate them. This happens because standard safety signals are often "indicator functions"—binary signals that only fire after a collision has occurred. This sparsity makes it incredibly difficult for an agent to learn the boundaries of safety. While humans can hand-craft "dense" cost functions (e.g., "penalize the agent as it gets closer to a wall"), doing so requires significant expert knowledge and tedious tuning for every new task.

Methods

The authors propose AutoCost, a bi-level optimization framework that automates the design of these cost functions.

  1. The Architecture: AutoCost adds an "intrinsic cost" ($c_{in}$) to the environment's "extrinsic cost" ($c_{ex}$). This intrinsic cost is represented by a small Multi-Layer Perceptron (MLP) that takes sensor data (like LiDAR) as input.
  2. The Inner Loop: A standard Safe RL agent is trained using the sum of extrinsic and intrinsic costs.
  3. The Outer Loop (Evolution): This is where the magic happens. AutoCost uses an Evolutionary Strategy to search for the best parameters for the intrinsic cost MLP.
  4. It starts with a population of 50 candidate cost functions.
  5. It evaluates them based on how well the resulting RL agent performs (prioritizing zero violations, then maximizing reward).
  6. The "fittest" cost functions are selected and mutated (via Gaussian noise and scaling) to create the next generation.

The result of this process is a specific cost function the authors call IC4SRL (Intrinsic Cost for Safe RL).

Datasets

The researchers evaluated AutoCost using the Safety Gym benchmark, a suite of environments designed specifically for testing safe exploration.

  • Tasks: "Goal" (navigate to a point) and "Push" (move a box to a target).
  • Constraints: "Hazards" (dangerous zones) and "Pillars" (physical obstacles).
  • Robots: They tested across three different robot models: Point (simple), Car (wheeled), and Doggo (a complex quadruped).

Key Insights and Ideas

  • The "Buffer Zone" Effect: Visualization of the evolved IC4SRL function shows that it naturally learns to assign high costs to states near hazards, not just the hazards themselves. It effectively creates a "safety margin" or a potential field that pushes the agent away from danger before a violation occurs.
  • Algorithm Sensitivity: The paper reveals that different RL algorithms react differently to the same cost function. A dense cost that helps CPO might actually hurt PPO-Lagrangian. AutoCost bypasses this by including multiple algorithms in its evaluation metric, finding a cost function that is robust across different learners.
  • Dense is Better than Sparse: The core "aha moment" is confirming that the failure of Safe RL isn't necessarily in the optimization algorithms themselves, but in the lack of informative, dense cost signals.

Key Results

  • Zero Violations: In the training environment (Goal-Hazard-Point), IC4SRL-augmented agents achieved zero constraint violations after convergence, whereas baseline agents continued to crash.
  • Impressive Generalization: The cost function evolved in a simple environment (Point robot) successfully transferred to the "Car" robot and the "Push" task, maintaining zero violations without any further tuning.
  • Performance Trade-off: While achieving zero violations usually results in slightly lower rewards (as the agent becomes more "cautious"), IC4SRL maintained a much better reward-to-safety balance than hand-tuned safety margins.

Limitations and Future Work

  • The "Doggo" Wall: The evolved cost function failed to generalize to the "Doggo" robot. The authors attribute this to the fact that the Doggo is so complex that the RL agent failed to learn any meaningful behavior, making the safety signal irrelevant.
  • Sensor Coupling: IC4SRL is currently coupled to the specific sensor inputs (LiDAR) of Safety Gym. Future work could explore more universal state representations.
  • Computational Cost: The evolutionary search is computationally expensive (taking about 54 hours in their setup), though the authors point out that once a good cost function is found, it can be reused instantly for new agents.
  • Real-World Deployment: The authors aim to move this framework from simulation to real-world robotics, where "zero-violation" isn't just a metric—it's a requirement for preventing expensive hardware damage.

Summary updated

Abstract

Safety is a critical hurdle that limits the application of deep reinforcement learning (RL) to real-world control tasks. To this end, constrained reinforcement learning leverages cost functions to improve safety in constrained Markov decision processes. However, such constrained RL methods fail to achieve zero violation even when the cost limit is zero. This paper analyzes the reason for such failure, which suggests that a proper cost function plays an important role in constrained RL. Inspired by the analysis, we propose AutoCost, a simple yet effective framework that automatically searches for cost functions that help constrained RL to achieve zero-violation performance. We validate the proposed method and the searched cost function on the safe RL benchmark Safety Gym. We compare the performance of augmented agents that use our cost function to provide additive intrinsic costs with baseline agents that use the same policy learners but with only extrinsic costs. Results show that the converged policies with intrinsic costs in all environments achieve zero constraint violation and comparable performance with baselines.
BibTeX
@inproceedings{he2023autocost,
  author = {Tairan He and Weiye Zhao and Changliu Liu},
  title = {{AutoCost: Evolving Intrinsic Cost for Zero-Violation Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26734},
}

Loading related content...

Figure