Imagined Violations: Training Safety Critics on Kinematic Counterfactuals

A robot learning by trial and error has two problems at once. It must not break itself or its surroundings while learning, and it cannot afford millions of real attempts. The usual way to keep a learning robot safe is a safety filter: a module that checks each proposed action and swaps it for a nearby safe one when needed. That keeps the robot intact, but it starves the learner. The learner never sees what its unsafe proposals would have done, so its estimate of "how dangerous is this action" stays poor. The safer the robot becomes, the fewer near-misses it sees, and the worse that estimate gets. This proposal exploits a fact particular to physical systems. Many of the constraints that matter are actuator torque limits, joint limits, acceleration and jerk bounds, and the turning geometry of a car. Their violation can be predicted from a simple analytic kinematic model without ever executing the action. We use that model to generate imagined violations: for states the robot actually visited, we label candidate actions it never took as "certainly leads to a violation within a few steps." Those labels train the safety estimate, while a filter guarantees that nothing unsafe is executed.

The key design choice is that the model is only ever allowed to say "unsafe," never "safe." Evidence that an action is safe comes only from real transitions. A wrong analytic model can therefore make the learned safety estimate more cautious, but it cannot make it more permissive. The claim is that this one-sided counterfactual labeling does three things. It removes the late-training safety degradation that otherwise requires deliberately unsafe exploration. It repairs the uninformative learning signal created by safety filters. And it does both with zero executed violations and fewer environment samples than current safe RL methods.

Core Claim and What Is New

Two recent results frame the problem. Breaking Safety Paradox with Feasible Dual Policy Iteration (FDPI) shows that as a policy becomes safe, its replay buffer runs out of violating transitions, which here means stored records of steps that broke a constraint. The feasibility critic is the learned estimator that predicts whether a state-action pair leads to violation, and it degrades when this data dries up, producing late "cost spikes." FDPI's fix is a second, KL-constrained "dual" policy that deliberately probes constraint boundaries. That motivates our work, but it constrains where FDPI can be used: on hardware, deliberate violations mean damage. Safe Reinforcement Learning using Action Projection shows that projecting unsafe actions onto the safe set maps many distinct actions to the same executed action. If the projection sits inside the policy, the policy gradient vanishes. If the filter sits in the environment, the critic "flat-lines" and cannot distinguish among the unsafe proposals. This is the second failure we target.

Our mechanism handles both with one data source. For a state $s$ drawn from real experience and a candidate action $a$, we roll an analytic kinematic model $f_{\text{kin}}$ forward for $H$ steps under a fixed fallback controller. If every rollout, under a set of model parameters inflated by a margin, enters the violation set, we record the counterfactual label $(s, a, 1)$. We train a reachability estimation function in the style of RESPO, which learns a Bellman-style estimate $\phi(s,a)$ of the probability of eventually violating. Real transitions use the standard recursive target. Counterfactual pairs use a one-sided target:

$$ \phi^{\text{target}}(s,a) = \max\big\{ \mathbb{1}_{\text{cert}}(s,a),\; \mathbb{1}_{s' \in S_v},\; \gamma\, \phi(s', \pi(s')) \big\}, $$

Here $\mathbb{1}_{\text{cert}}(s,a)$ is 1 when the kinematic model certifies violation within $H$ steps and 0 otherwise, $S_v$ is the violation set, $s'$ is the real next state (this term is absent for purely counterfactual pairs), $\gamma$ is the discount, and $\pi$ is the current policy. Because the certificate enters only through a maximum, an incorrect model can raise $\phi$ but never lower it. The actor is trained to maximize return subject to $\phi(s,\pi(s)) \le \epsilon$, with $\epsilon$ the tolerated violation probability. The constraint is evaluated on the unprojected action, so aliased proposals receive distinct, graded penalties. A safety filter remains in the execution loop and guarantees that nothing unsafe reaches the actuators. Its purpose shifts from patching a policy that never learns the boundary to serving as a backstop whose intervention rate should fall toward zero.

The closest prior work does not combine these pieces. Kochdumper et al. project actions using reachable sets computed with polynomial zonotopes, a compact way of over-approximating every state the system could reach. They enforce safety but leave the learner exposed to aliasing. Their reachability computation scales as $O(n^5)$ in state dimension, while we need only cheap forward simulation of a low-dimensional kinematic model to produce labels. VELM learns symbolic dynamics for shielding but is unprotected at cold start, whereas the analytic kinematic model is available from the first step. SafeDreamer and SOOPER use learned world models whose cost predictions can hallucinate safety in unvisited regions. Our model is analytic and partial, and it is restricted to asserting danger. Hierarchical feasibility work such as Wen et al. penalizes unreachable waypoints through a reward term and a 7-stage curriculum. That gives no hard guarantee during training, and we aim to remove the curriculum.

Research Program

The work has three parts. The first is the labeling engine. For each robot class we write a deliberately simplified model, never the simulator itself. For the Point and Car agents this is unicycle or bicycle kinematics with velocity and acceleration bounds. For the arm tasks it is rigid-body forward kinematics with joint, torque and singularity limits, following the limits used in Zhang et al.. For driving it is a kinematic bicycle model. Candidate actions are drawn both near the current policy action and uniformly over the action box. This concentrates labels near the decision boundary that FDPI's dual policy is designed to probe.

The second part is the learner. It is an off-policy actor-critic built on the RESPO-style reachability critic, with a constraint multiplier stabilized by the PID controller that Safety-Gymnasium found necessary to prevent oscillating Lagrange multipliers. Training includes counterfactual pairs at a fixed ratio to real pairs.

The third part is analysis. We will measure how often counterfactual labels disagree with real outcomes, which tells us how conservative the model is. We will measure how the filter's intervention rate evolves. And we will test whether late-training cost spikes disappear when the filter is removed at evaluation.

Scope is a real limitation. Contact-rich costs and interactions with other agents are not covered by a kinematic model. For those constraints our method falls back to ordinary real-data critic learning, and we will report results separately for constraints the model covers and constraints it does not.

Baselines

The naive reference point is PPO with the Lagrangian method (PPO-Lag) on the standard cost signal, with no filter. On velocity tasks, Safety-Gymnasium reports that PPO-Lag cut violations by 98% relative to PPO at a 45% reward drop. Any method claiming sample efficiency must at least match this baseline's return while executing far fewer violations.

The strongest competitor on the safety-paradox question is FDPI with a SAC backbone (SAC-FDPI). The file reports that it reached near-zero violations across 14 environments while matching or exceeding CPO, PPO-Lag, SAC-Lag and SAC-FPI. It also kept roughly 10x more violating samples in its buffer than SAC-FPI, whose violation ratio fell below 1% late in training. We run FDPI in two modes. In the first, its dual policy is allowed to execute violations, as published. In the second, the same execution filter we use is applied, which represents a hardware-legal FDPI. Beating FDPI means matching its final return and late-training cost in the first mode with zero executed violations, and clearly exceeding it in the second mode.

The strongest competitor on sample efficiency is ESPO. It reports reaching its performance level with 25–29% fewer samples, and specifically needed 5.7 million samples on SafetyReacher-v4 against 8 million for baseline methods. We will count samples to the return level ESPO reports on SafetyReacher-v4 and SafetyHumanoidStandup-v4. Beating ESPO means reaching that level in fewer real samples while ESPO is still incurring early-training violations.

The strongest filter baseline is the Kochdumper reachability projection shield with an unmodified learner. On CommonRoad it reduced collisions from 10 to zero across 2,000 scenarios at 0.043 seconds per solve. We use the same shield as our execution backstop, so the comparison isolates the learning signal. We also include SafeDreamer as the learned-world-model alternative and RESPO without counterfactuals.

All methods receive the same environment-step budget, network sizes and a matched hyperparameter search of equal trials per method, and are evaluated on 5 or more seeds. Counterfactual model rollouts are reported as wall-clock compute but not counted as environment samples, and we report both.

Three ablations isolate the key ingredient. The first symmetric-labels variant lets the model also assert "safe," which tests whether the one-sided design matters. The second replaces the analytic model with a learned dynamics ensemble, which tests whether analytic structure matters. The third uses a parameter-perturbed analytic model at 10–40% error, which tests robustness to model mismatch.

Why not just…?

Why not just penalize the distance between the proposed and projected action? This is the most serious threat. It directly gives aliased actions distinct gradients and needs no model. Its weakness is that it only measures the immediate correction. An action can require no projection now and still commit the robot to an unavoidable violation two steps later, for example a car entering a turn too fast. The multi-step reachability label captures that case and the penalty does not. Penalty magnitudes also carry the fragility documented by Quanyi Li et al.: too small and violations are tolerated, too large and the policy freezes. We run this baseline with its own tuning budget. If it matches our method on intervention rate and late-training cost spikes, the multi-step counterfactual labels are not earning their place, and we would report that.

Why not just run FDPI in simulation and transfer the policy? Transfer would carry the sim-to-real gap that Amato et al. observed for unconstrained policies. It would also leave the critic unable to keep learning on hardware, which is precisely when the safety paradox appears. The filtered-FDPI comparison above measures what remains of FDPI's benefit once its violations are prohibited.

Why not just use a learned world model as SafeDreamer or SOOPER do? Learned models are least accurate in exactly the regions a safe policy never visits, which is where boundary labels are needed. SOOPER additionally assumes its fallback prior is executable from every state. The learned-ensemble ablation tests this alternative directly. Analytic structure should win on constraints the kinematic model covers. Learned models may win on contact-driven costs, and we will report both outcomes.

Proposed Experiments

We run Safety-Gymnasium navigation tasks (PointGoal, CarGoal, PointButton) and velocity tasks, the Safety-MuJoCo tasks used by ESPO (SafetyReacher-v4, SafetyHumanoidStandup-v4, SafetyAntVelocity-v1), and CommonRoad driving with the Kochdumper shield as backstop. The metrics are executed violations during training, filter intervention rate over time, real samples to the reference return, final return and cost, and late-training cost measured with the filter removed at evaluation.

Three outcomes decide the proposal. The first concerns the safety paradox. With the filter removed at evaluation, critics trained without counterfactual labels should show the late cost spikes FDPI describes, and ours should not, while executed training violations stay at zero. The second concerns the learning signal. The intervention rate should fall steadily under our method and stay flat under shield-plus-RESPO, which would indicate that the flat-critic failure has been repaired. The third concerns model error. Performance should degrade gracefully, toward a more cautious policy, under perturbed models with one-sided labels, and should fail unsafely under symmetric labels. If the first outcome fails, the safety-paradox claim is wrong. If the projection-distance penalty matches us on the second, the contribution narrows to the one-sided labeling result alone.