Barrier Rewards Are Mostly Potentials: Robust Barrier Costs for Learning Under Model Uncertainty

Reinforcement learning agents for robots usually get a hand-written reward with extra penalty terms that keep joints, velocities or positions inside safe limits, and tuning those penalty weights is a well-known chore. A recent shortcut borrows from control theory. A barrier function $h(s)$ is a score that is positive inside the safe region and zero on its edge. The shortcut adds its rate of change, plus a multiple of $h$ itself, to the reward. That removed most joint-limit tuning and cut actuation energy roughly in half in Barrier Functions Inspired Reward Shaping for Reinforcement Learning.

We point out that this reward has a hidden structure that matters a great deal once the robot's model or sensors are uncertain. The rate-of-change part is computed by finite differences of observed states. Under a discounted objective, almost all of it is potential-based shaping, meaning a reward term of the form (future potential minus current potential). Such terms provably cannot change which policy is optimal. What remains is a plain state penalty proportional to $h$, with a weight that depends on the control period and the discount. So barrier shaping speeds up learning but protects safety only as a soft penalty. Its rate term also injects sensor noise that grows as the control period shrinks. We propose to keep the harmless potential part for speed and to move the safety-relevant part out of the reward entirely. It becomes a constraint cost built from a robust barrier condition, which penalizes approaching the boundary faster than the measured uncertainty allows. That cost is then enforced with a smoothed-log-barrier constrained optimizer, so no penalty weight has to be hand-tuned.

Core Claim

Write $\beta$ for the RL discount, $\Delta t$ for the control period, and $c>0$ for the barrier gain. The finite-difference barrier reward then decomposes exactly as

$$ \frac{h(s')-h(s)}{\Delta t} + c\,h(s) \;=\; \underbrace{\Big[\beta\,\Phi(s') - \Phi(s)\Big]}_{\text{policy-invariant}} \;+\; \frac{1-\beta}{\Delta t}\,h(s') + c\,h(s), \qquad \Phi = h/\Delta t . $$

The bracketed term telescopes along any trajectory. This holds even when $h$ is evaluated on noisy observations, because the potential is still a function of what the agent sees. So the only part that shapes the optimal policy is a state reward of effective weight $\kappa \approx c + (1-\beta)/\Delta t$. With $\beta=0.99$ and $\Delta t=0.01$, that hidden weight is about 1 and can dominate $c$. Meanwhile the per-step noise added to temporal-difference targets scales like $\sigma\|\nabla h\|/\Delta t$, where $\sigma$ is the sensor noise level.

This yields three testable predictions. First, replacing barrier shaping by the pure state penalty $\kappa h$ should reproduce its final returns and violation rates, though not its learning speed. Second, under parameter drift barrier shaping should degrade exactly as a soft penalty does. Third, under observation noise its speed advantage should shrink as $\sigma/\Delta t$ grows.

The method we build on this analysis is a robust barrier cost:

$$ C_t = \max\!\Big(0,\; m - \tfrac{h(o_{t+1})-h(o_t)}{\Delta t} - \alpha\,h(o_t)\Big), $$

Here $o_t$ is the noisy observation, $\alpha>0$ is the barrier decay rate, and $m\ge 0$ is a robustness margin. The cost is nonzero only when the agent closes on the boundary faster than the barrier condition permits, inflated by $m$. Because $C_t$ is a nonlinear function of the transition, it is not a potential and does change the optimum. Because it fires before the boundary is reached, it is dense, unlike a cost that only counts actual violations.

The margin $m$ is not tuned per task. It is set to a fixed quantile (95% everywhere) of the residual $\frac{\Delta h}{\Delta t}+\alpha h$, measured on calibration rollouts across the training uncertainty set. This mirrors how distributionally robust barriers size their margin from empirical sensor samples (Sensor-Based Distributionally Robust Control for Safe Robot Navigation in Dynamic Environments). The policy is trained on the task reward plus the potential term $\beta\Phi(s')-\Phi(s)$. It is subject to an expected discounted $C_t$ budget near zero, enforced with the linear smoothed log barrier of Constrained Reinforcement Learning with Smoothed Log Barrier Function.

Relation to Prior Work

Three papers motivate the gap. Barrier reward shaping (Nilaksh et al.) adds $\dot h + c h$ to the reward, estimated model-free, and reports faster convergence and lower energy. It was evaluated with clean states and nominal dynamics, and it offers no constraint guarantee. We reuse its finite-difference barrier signal but split it into its policy-invariant and policy-changing parts. Robustified barrier functions (Safety Under State Uncertainty: Robustifying Control Barrier Functions) add a robustness term to the barrier inequality, giving exact safety under small errors and bounded inflation under large ones. The distributionally robust barrier (Sensor-Based Distributionally Robust Control) sizes its constraint from empirical noise samples. Both are runtime filters that need a control-affine model, and the latter assumes relative degree 1. We take their idea of a data-sized margin but apply it as a training-time cost, which needs no dynamics model.

The method's machinery comes from CSAC-LB (Constrained Reinforcement Learning with Smoothed Log Barrier Function). It replaces penalty and Lagrangian tuning with a log barrier that turns linear past the boundary, plus a pessimistic cost critic. Its cost signals are the task's native costs, and it was never tested under parameter perturbation or sensor noise. We change the cost it optimizes, not the optimizer.

Two lines constrain the design. Differentiable Safe Controller Design through Control Barrier Functions shows that post-hoc barrier filters are myopic and hurt long-horizon return. It fixes this with differentiable safety layers that assume known dynamics and known linear constraints. We avoid any known model by keeping safety in the training objective. Learning a Better Control Barrier Function Under Uncertain Dynamics learns $\dot h$ directly rather than $\dot x$, to tolerate large mass errors. That supports our choice to work with the barrier's derivative as the quantity of interest rather than full-state dynamics. Finally, Safe Inverse Reinforcement Learning via Control Barrier Function uses a related barrier-derivative loss inside imitation learning and reports that conservative barriers trap the agent. This is a direct warning that an oversized margin $m$ will cost reachability, and our margin sweep is designed to measure it.

Research Program

The work has three parts.

The first is analytic. We state the decomposition above precisely, including the $h(s)$ versus $h(s')$ offset and episode-termination effects, and derive the variance contributed by observation noise to the critic's targets.

The second is a diagnostic study of existing barrier shaping on the robust locomotion tasks of RRLS, using its parameter-perturbation wrappers (SuReLI/RRLS). We add Nilaksh-style joint-limit barriers to Ant, HalfCheetah, Hopper and Walker2d, and inject Gaussian observation noise at several levels and control rates.

The third is the method itself, built on the CSAC-LB codebase with the robust barrier cost replacing the native cost and the potential term added to the reward. We also evaluate on Safety-Gymnasium velocity-limited locomotion (Safety Gymnasium), where the barrier is $h = v_{\max} - v$ and the native velocity cost gives a ground-truth violation count.

Baselines

The naive reference is the plain task reward trained with SAC under domain randomization over the RRLS parameter ranges, with no safety terms. It shows how often limits are hit when nobody engineers anything.

The direct competitor from reward engineering is Nilaksh-style barrier shaping (Barrier Functions Inspired Reward Shaping). We run it with the same domain randomization and observation noise, and with its barrier gain grid-searched on the nominal environment using the same tuning budget we give our method's single quantile. Its reported result is a 1.56× faster convergence and about 49% of baseline actuation energy on Humanoid. No reported result found on Lacuna for barrier shaping under parameter perturbation or sensor noise. Beating it means fewer worst-case violations at equal or better nominal return.

The strongest existing constrained method is CSAC-LB with the native violation cost under the same domain randomization. It reports roughly 24% higher returns than the next best safe baseline and near-zero violations in 9 of 10 environments (Constrained Reinforcement Learning with Smoothed Log Barrier Function), in unperturbed simulation. It is the strongest competitor because it shares our optimizer, so any gain must come from the cost. Beating it means a lower violation rate under the worst-case RRLS perturbation and under noise, at no more than a 5% loss in nominal return, across 10 seeds with matched environment steps.

We also run SAC-Lagrangian and PPO-Lagrangian from SafePO. The reference points for them are average costs up to 700 for SAC-Lagrangian (CSAC-LB) and, for PPO-Lagrangian, a 98% violation reduction alongside a 45% reward drop (Safety Gymnasium). A further baseline is a post-hoc robust barrier filter applied to the naive policy. It uses a nominal model with an inflated margin, built with CBFkit.

Three ablations isolate the ingredients. Setting $m=0$ removes robustness. Dropping the potential term removes the speed component. Moving $C_t$ into the reward with a hand-tuned weight instead of the constraint tests whether the constraint machinery is what removes tuning.

Why not just…?

Why not just keep barrier reward shaping and add domain randomization? This is the most serious threat, and we treat it as a primary baseline rather than a straw man. Our decomposition predicts that its safety effect equals a soft state penalty of weight $\kappa$. RRLS reports that domain randomization helps average-case but not worst-case performance. If barrier shaping plus randomization matches our worst-case violation rate, the robust cost adds nothing, and we will report that.

Why not just low-pass filter the observations before taking finite differences? Filtering reduces the $\sigma/\Delta t$ noise but introduces lag. That lag makes the rate estimate late exactly when the agent is closing on a limit. More to the point, the decomposition shows that noise in the potential part never changed the optimum, only learning speed. Filtering therefore cannot fix the drift problem, which lives in the soft-penalty part. We include a filtered barrier-shaping variant to measure this directly.

Why not just use CSAC-LB or a Lagrangian method on the native violation cost? Native costs fire only after a violation. Under parameter drift, a policy trained to sit just inside the limit on nominal dynamics crosses it when mass or friction shift. The robust barrier cost penalizes the approach rate with a margin sized from the uncertainty set, which should produce a buffer. If CSAC-LB with native costs is already near-zero under worst-case perturbation, our contribution reduces to the diagnostic analysis, and the matched comparison will say so.

Why not just tune the penalty weight by hand? CSAC-LB documents that hand-set penalties are either too weak or too conservative. Our reward-hinge ablation measures how sensitive the outcome is to that weight compared with our single fixed quantile.

Proposed Experiments

The first decisive check tests the decomposition. We train with $r_{\text{task}}+\kappa h$ and with full barrier shaping, matched by seed. If the decomposition is right, final return, violation rate and energy should agree within seed variance while learning curves differ. A significant gap in final behavior would falsify the analysis.

The second check sweeps observation noise $\sigma$ and control period $\Delta t$. We expect the speedup from barrier shaping to fall as $\sigma/\Delta t$ grows, while our method's violation rate stays flat because $m$ grows with the measured residual spread.

The third check evaluates every method under the RRLS worst-case adversarial perturbations, including torso and leg mass changes on Ant. Metrics are joint-limit or velocity violation rate per 1,000 steps, worst-case and nominal return, and actuation energy.

The fourth check sweeps the margin quantile from 50% to 99% to trace the safety–reachability curve that Safe Inverse RL via CBF warns about. If no quantile beats CSAC-LB with native costs on worst-case violations at matched nominal return, the method claim fails.