In traditional safe reinforcement learning (RL), safety is usually treated as a budget. As long as an agent doesn't crash too often on average, it is considered "safe." This is typically modeled using Constrained Markov Decision Processes (CMDPs), where the goal is to maximize rewards subject to a cumulative cost threshold.
However, as Ming Shi, Yingbin Liang, and Ness Shroff argue in A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraints, an average is not good enough for high-stakes environments. A power grid operator cannot "average out" a blackout, and an autonomous vehicle cannot "offset" a collision by driving safely for the rest of the day. These scenarios require instantaneous hard constraints: safety requirements that must be satisfied at every single time step, without exception.
The Problem with Averages
The core intuition behind this research direction is that cumulative constraints allow for "safety trading." In a standard CMDP, an agent might take a high-risk action early in an episode if it believes it can accumulate enough "safety margin" later to keep the total expected cost low.
Haitong Ma, Shengbo Eben Li, and colleagues point out in Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety that global penalties fail to protect specific, hazardous regions of the state space. If a penalty is applied globally, the agent might still enter a catastrophic state if the reward for doing so is high enough or if the state is rare enough to not significantly impact the average. To solve this, researchers are moving toward state-wise or pointwise safety, where the constraint is local to the current moment.
Enforcing Safety at the Control Level
How do you ensure an agent never violates a constraint while it is still learning? One dominant approach is to modify the action space itself.
In Robot Reinforcement Learning on the Constraint Manifold, Puze Liu and Jan Peters introduce ATACOM. Instead of letting the agent pick any action and then penalizing it for being unsafe, they project the agent's exploration into the tangent space of the constraint manifold. This ensures that every action the robot takes is physically and safely feasible by construction, effectively turning a constrained problem into an unconstrained one within a "safe" subspace.
Similarly, Yixuan Wang and Zhilu Wang propose using Barrier Functions in Enforcing Hard Constraints with Soft Barriers. These functions act like a magnetic field that gets infinitely stronger as the agent approaches an unsafe boundary, preventing the agent from ever crossing into hazardous territory, even in unknown stochastic environments.
The Paradox of Aggressive Exploration
A significant challenge in this field is the "cold start" problem: how can an agent know what is safe if it hasn't explored the environment yet?
In Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration, Honghao Wei and Lei Ying introduce a counter-intuitive strategy. They argue that to satisfy hard constraints later, the agent must engage in aggressive exploration early on to map out the safety boundaries. By using a two-phase approach—first identifying the safe set and then optimizing within it—algorithms like NCS-LSVI, developed by Amirhossein Roknilamouki and Arnob Ghosh, can achieve zero safety violations with high probability while still maintaining near-optimal learning efficiency (Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints).
Beyond Average Regret
The shift toward instantaneous constraints also changes how we measure success. In standard RL, we minimize "regret" (the difference between the agent and the optimal policy). In hard-constrained RL, we must also minimize violation regret.
As Alberto Marchesi and Nicola Gatti demonstrate in Learning Adversarial MDPs with Stochastic Hard Constraints, the "cost of safety" is an unavoidable dependence on the environment's feasibility margin. If the safe path is very narrow, the agent must be more conservative, which naturally slows down its ability to maximize rewards.
By prioritizing per-step operational security over cumulative averages, this research direction—summarized in the survey by Weiye Zhao and Changliu Liu (State-wise Safe Reinforcement Learning: A Survey)—is bridging the gap between theoretical RL and the uncompromising safety requirements of real-world robotics, healthcare, and infrastructure.