A robot learning by trial and error has to learn two things: how to finish the task, and where the dangerous states are. The second is much harder to learn from data, because a good learner rarely crashes, and each crash is the only direct evidence it gets about a hazard. We propose to build the symmetries of the physical world into the network that predicts future safety cost, which we call the safety critic. If the scene is rotated, the danger of a state does not change, so the critic's prediction should not change either. With this built in, one collision near a hazard seen at one orientation teaches the agent about that hazard at every rotated and mirrored orientation. The aim is to cut the number of physical violations an agent needs during training, which is the quantity safe exploration actually cares about, rather than only speeding up reward learning.
The claim in two sentences: equivariant networks (networks whose outputs transform predictably when the input is rotated or reflected) have so far been used to speed up reward learning. Their largest payoff should come from the safety side, where data is scarcest, and this should be measurable as fewer training-time violations and fewer late-training cost spikes, scaling with the size of the symmetry group.
Core Claim
Equivariant reinforcement learning shows that encoding rotations and reflections into the network multiplies the value of each sample. SO(2)-Equivariant Reinforcement Learning builds rotation-equivariant actors and rotation-invariant critics for Soft Actor-Critic (SAC) and reaches peak Block Pulling performance in roughly 2,500 steps, against more than 10,000 for augmentation baselines. Equivariant Offline Reinforcement Learning reports roughly doubled data efficiency from a handful of demonstrations. Both papers optimize reward only. Safe RL, meanwhile, keeps hitting a data problem on the cost side. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that as a policy becomes safe, violations fall below 1% of the replay buffer. The cost estimator then degrades and the policy suffers sudden cost spikes. Their fix, FDPI, adds a second "dual" policy that deliberately seeks violations near the safe policy.
Our hypothesis is that the cost critic, not the reward critic, is where symmetry matters most, because it is the network starved of data. Take a constrained Markov decision process, meaning a standard RL problem plus a cost signal $c(s,a)$ whose expected discounted sum must stay below a budget $\epsilon$. Suppose a group $G$ of rotations and reflections leaves the dynamics, the reward, and the cost unchanged. Then the cost critic must satisfy
$$Q_c(g\cdot s,\, g\cdot a) = Q_c(s,a) \quad \text{for all } g\in G,$$
and the same argument that makes the reward critic invariant in the SO(2) paper applies directly. When $G$ acts freely on states near hazards, each recorded violation then constrains $|G|$ distinct state-action pairs. FDPI's argument says the cost estimator's error is governed by how many distinct boundary-crossing transitions it has seen. So the number of physical violations needed to reach a given estimation error should shrink by up to a factor of $|G|$. This is the quantitative prediction we test. We expect the factor to hold on scenes whose symmetry is intact and to shrink where workspace walls or fixed fixtures break the symmetry.
The closest existing work is Deep Equivariant Multi-Agent Control Barrier Functions. That paper makes learned safety certificates invariant and controllers equivariant for quadrotor swarms. It relies on control barrier functions, which require known control-affine dynamics and state derivatives. Our setting is model-free and image-based: an unknown simulator, a trajectory-level cost budget, and a safety critic learned from replay. That paper motivates the invariance argument; it does not supply a method we can reuse.
Research Program
The base agent is SAC with a Lagrange multiplier, a learned penalty weight that rises when cost exceeds the budget and falls otherwise. We stabilize the multiplier with the PID update that Safety Gymnasium reports reduces dual oscillation. The networks follow the SO(2) paper's design: steerable convolutions over a cyclic rotation group $C_n$, built with the same E2CNN machinery. There are three networks: an equivariant actor, an invariant reward critic, and an invariant cost critic.
Real scenes rarely have exact symmetry; a table edge or robot base breaks it. We therefore add a small unconstrained residual head to the cost critic only. Its output is scaled by a learned gate that is penalized toward zero. The cost critic can then represent symmetry-breaking hazards such as walls without discarding the shared orbit structure elsewhere. Measuring how far the gate opens, task by task, tells us how much symmetry the safety problem actually has.
The second component tests whether equivariance and active probing are complementary. We combine the equivariant cost critic with FDPI's dual policy and its importance-weighted replay. The prediction is that FDPI then needs fewer deliberate violations to keep its cost estimator sharp, because each probe is shared across the group.
We use two testbeds chosen for their symmetry size. The main one is a safe variant of BulletArm built on the top-down, workspace-frame tasks where $C_8$ equivariance holds exactly in the SO(2) paper: Block Pulling, Object Picking, and Drawer Opening. We add randomly placed fragile objects and forbidden table regions. The cost is 1 whenever the gripper contacts a fragile object or enters a forbidden zone. These cost functions are new and are part of what we build. The second testbed is Safety-Gymnasium PointGoal1, ButtonGoal1, and CarGoal1. There the robot sees the world egocentrically and is steered in its own body frame, so only left-right reflection ($|G|=2$) holds. We expect modest gains there and report them as the weak-symmetry case.
Baselines
The naive reference is non-equivariant SAC-Lagrangian with PID multiplier updates and the same encoder depth, trained on the same seeds and step budget. It establishes how many violations an agent without symmetry accumulates.
The strongest existing safe-exploration method is SAC-FDPI from Breaking Safety Paradox with Feasible Dual Policy Iteration. It is the only method in the file that directly addresses cost-estimator starvation. It reports near-zero violations across 14 Safety-Gymnasium environments while matching or exceeding the reward of CPO, PPO-Lag, SAC-Lag, and SAC-FPI. In the final 10% of training it keeps roughly 10x the violation-sample ratio of SAC-FPI, whose ratio falls below 1%. The file reports no scalar return or cost values for FDPI, and no reported result was found on Lacuna for any method on safe BulletArm variants, since they do not yet exist. We will therefore rerun FDPI ourselves on both testbeds with its published structure, our network sizes, and an identical tuning budget for both methods. On Safety-Gymnasium we use the same budget of 20 configurations over learning rate, dual-policy KL weight, and cost limit, and check that the PointGoal1 results qualitatively reproduce its claims before trusting the comparison. Beating FDPI means at most half its cumulative training-time cost at equal or higher final return on the $C_8$ tasks. On the $|G|=2$ tasks, matching FDPI with no dual policy would already count as a positive result.
On the reward side, the reference is Equivariant SAC from SO(2)-Equivariant Reinforcement Learning. It reports peak Block Pulling performance at about 2,500 steps against more than 10,000 for RAD and DrQ. We run it with a reward penalty $r-\lambda c$ swept over five values of $\lambda$ and report the best setting. This shows what symmetry buys without any constraint machinery.
The key ablation crosses equivariance between the two halves of the agent. One variant has an equivariant actor and reward critic but a plain cost critic. The other has a plain actor and reward critic but an equivariant cost critic. All variants use identical parameter counts, matched by widening the plain networks. If the claim is right, the second variant recovers most of the safety gain and the first recovers most of the reward gain but little safety. A further ablation removes the symmetry-breaking residual to isolate its contribution near walls.
Why not just…?
Why not just augment the replay buffer with rotated and reflected copies of every transition, cost included? This is the most serious threat. It gives a plain network the same "one crash, many orientations" data multiplication without steerable layers. The reward-side evidence favors architecture. In the SO(2) paper, RAD and DrQ needed over 10,000 steps where Equivariant SAC needed about 2,500, and they failed outright on Object Picking and Drawer Opening. Rare-event cost estimation is a different regime, though, and full-orbit augmentation of a handful of violations may suffice. We therefore run the strongest version: every transition augmented over all of $G$, with the cost critic's update ratio matched. If it matches the equivariant cost critic on training-time violations, we will report that the gain comes from orbit sharing itself, not from the architecture, and recommend augmentation for safety critics because it is simpler.
Why not just use FDPI, since active probing already fixes the data starvation? FDPI fixes starvation by generating more violations, and in a physical setting each of those is a real collision. Equivariance tackles the same estimation error from the other side, by making each violation count $|G|$ times. The two are not exclusive, which is why we test their combination. The measurement that settles it is cumulative training cost at matched final cost-estimator error. If plain FDPI reaches that error with no more violations than equivariant FDPI, symmetry adds nothing on the safety side.
Why not just add a collision penalty to Equivariant SAC and skip constrained optimization? Safety Gymnasium and Recovery RL document the trade-off: small penalties tolerate violations, and large ones freeze the agent. Recovery RL reports 2 to 20 times better task-safety trade-offs than penalty baselines. Symmetry does not remove this sensitivity to the penalty weight, but it could shift where the good weight lies. The $\lambda$ sweep shows whether any fixed penalty matches the constrained agent's return at the same budget.
Proposed Experiments
All methods run for 10 seeds. Each BulletArm task gets 20,000 steps, following the budget scale of the SO(2) paper. Safety-Gymnasium tasks get 2 million steps. We report final episodic return and final episodic cost against the budget. We also report three exploration-specific measures: cumulative training-time violations, the fraction of violation samples in the replay buffer over training, and the number and size of late-training cost spikes, defined as episodes whose cost exceeds twice the budget after the cost has first converged.
The first decisive check is the scaling prediction. On safe Block Pulling and Object Picking we train the cost critic with groups $C_1$, $C_2$, $C_4$, and $C_8$ while holding everything else fixed. We plot cumulative training-time violations against $|G|$. A roughly $1/|G|$ decline on symmetric layouts confirms the orbit-sharing mechanism. A flat curve refutes it. Repeating the experiment on layouts with a hazard fixed against a wall should show the decline flattening and the residual gate opening.
The second decisive check is the actor-versus-critic ablation described above, which locates where symmetry pays. The third is a held-out orientation test. We train with hazards confined to one quadrant of the table and evaluate with hazards in the others. The invariant cost critic should keep near-budget cost zero-shot. The plain and augmented critics should not, mirroring the zero-shot transfer result the SO(2) paper reports for reward.
On Safety-Gymnasium we expect smaller but nonzero improvement in cost spikes from reflection alone. A null result there, together with a clear result at $C_8$, would itself support the claim that the benefit tracks group size.
References
- $\mathrm{SO}(2)$-Equivariant Reinforcement Learning
- Equivariant Offline Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Deep Equivariant Multi-Agent Control Barrier Functions
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Framework
- Recovery RL: Safe Reinforcement Learning With Learned Recovery Zones