A robot learning by trial and error has to find out where "too fast", "too close" or "too much force" begins, and it usually learns this by crossing the line. Safe reinforcement learning (RL) methods keep the final policy within a safety budget, yet still rack up thousands of violations during training. They also suffer from a known trap: once the agent becomes careful, it stops seeing near-failures, and its learned estimate of danger decays. Extremum seeking control offers a different way to learn about the boundary. It is a classical control technique that adds a tiny sinusoidal wiggle (a "dither") to a system's input and reads the matching ripple in a measured output to recover the local slope, with no model of the plant. We propose to use small wiggles of this kind on the real executed actions to measure, online and on the safe side of the line, how sensitive a physically measurable constraint signal (speed, distance to a hazard, contact strain) is to each action. Those measured slopes then serve two purposes. They drive a runtime filter that bends exploratory actions away from the boundary before it is crossed. They also fill the replay buffer with dense, labeled near-boundary data, which feeds the learned cost estimate without anyone ever having to crash.
The opportunity is that each probing step yields slope information about the true constraint rather than only a point sample. This should cut both the number of training-time violations and the number of samples needed to reach a boundary-hugging, high-reward policy. The setting where this matters most is hardware. There, the strongest existing remedy for the "careful agent stops learning about danger" problem deliberately generates violations, which is unacceptable on a real machine.
Core Claim
The closest prior work, Extremum-Seeking Action Selection (ESA), adds a dither to sampled actions and climbs the learned reward critic $Q(s,a)$. It has no safety mechanism at all. The obvious extension adds a second, cost critic and projects along its level set, and a skeptical reviewer would rightly call that redundant. Dithering a neural network that sits in software only gives a noisy finite-difference estimate of a gradient that automatic differentiation computes exactly, and it inherits whatever errors the critic has near a boundary it has rarely seen. We therefore move the dither out of the critic and into the plant. Our claim is that measured, plant-level constraint sensitivities are most accurate exactly where learned cost critics are weakest: early in training and late in training, once the agent has become safe. A dither-measured safety filter with a dither amplitude that shrinks near the boundary should therefore achieve fewer cumulative training violations than Lagrangian, boundary-seeking, and dual-policy methods. It should do so at equal or better sample efficiency and without ever needing a deliberate violation.
Research Program
Assume the task exposes an instantaneous constraint signal $g_t$, a measured scalar where $g_t \le 0$ is safe and the per-step cost is the indicator that $g_t > 0$. Examples are forward speed minus the speed limit in the Safety-Gymnasium velocity tasks, negative distance-to-hazard from the lidar in the navigation tasks, and peak tactile strain minus a threshold in insertion. The base learner is an off-policy safe RL agent: SAC-Lagrangian from the SafePO library of Safety-Gymnasium. We chose an off-policy learner because executed (filtered) actions can be stored and reused, which addresses the on-policy sample cap that limits FOCOPS.
At every control step the executed action is the policy action plus a correction and a dither, $a_t = a^\pi_t + \Delta_t + A_t \odot \sin(\omega t)$. Each action dimension gets its own frequency, chosen so that sums and differences of frequencies do not collide; this is the standard multi-variable extremum seeking construction. Multiplying the measured ripple in $g$ by each sinusoid and low-pass filtering gives a running estimate $\hat J_t$. This is the sensitivity of the constraint signal $k$ steps ahead to the current action. We use a short horizon $k$ because speed responds to torque within a step or two, but position-type constraints respond with lag. We calibrate the phase lag per task, as classical extremum seeking does.
Two rules use this estimate. The first sets the dither amplitude so that the predicted excursion of the wiggle cannot reach the boundary:
$$A_t = \min\!\Big(A_{\max},\; \kappa \,\frac{\max(0,-g_t - m)}{\lVert \hat J_t \rVert + \eta}\Big),$$
where $m$ is a safety margin, $\kappa<1$ is a scaling factor, and $\eta$ prevents division by zero. The second is a one-constraint projection. It removes only the component of the policy action predicted to push $g$ past $-m$:
$$\Delta_t = -\frac{\big[\,g_t + \hat J_t^\top (a^\pi_t - \bar a_t) + m\,\big]_+}{\lVert \hat J_t\rVert^2 + \eta}\,\hat J_t,$$
where $\bar a_t$ is the filtered baseline action around which $\hat J_t$ was measured.
Shrinking the amplitude near the boundary weakens the slope signal. Averaging theory for demodulated estimates, of the kind analysed in Large Gain Stability of Extremum Seeking, suggests that the variance of the estimate scales as roughly $\sigma^2/(A_t^2 N)$, where $N$ is the number of dither periods averaged and $\sigma^2$ is the measurement noise. The design question is therefore how the margin $m$, the averaging window, and the noise trade off. Our first analysis will be an empirical map of slope-estimate error against distance-to-boundary.
The second use of the dither is data. Every dithered step near the boundary is stored with its continuous $g$ value and its measured $\hat J$. The cost critic is trained on this denser band of near-boundary transitions, with an optional gradient-matching term that we will ablate. The goal is to test whether this keeps the critic calibrated late in training. The failure we are guarding against is the safety paradox documented by Feasible Dual Policy Iteration (FDPI): a safe agent stops seeing violations and its feasibility estimate collapses into late cost spikes.
FDPI cures this with an adversarial policy that seeks violations, an approach that is not viable on hardware. We instead probe from the safe side, in the spirit of Source Seeking with Convex Obstacles. That work runs extremum seeking on a navigation potential with online repulsive barriers but learns no policy and is limited to convex obstacles. Here the learned policy supplies global behaviour, which removes the local-trapping weakness of pure extremum seeking, and the dither supplies local boundary truth.
This design inherits two known limits. Time-scale separation between plant, dither, and filter is required, and multi-dimensional actions need many dither periods per estimate; for example, the Ant agent has 8 action dimensions. We will therefore also test dithering only along the top few directions of the cost critic's action gradient, which reduces the dimension that must be resolved.
Baselines
The naive reference is SAC-Lagrangian and PPO-Lagrangian from SafePO, run unmodified. Safety-Gymnasium reports that on velocity-constrained locomotion PPO-Lag achieved a 98% reduction in safety violations compared to standard PPO, with a 45% drop in total reward. That 45% reward penalty is the conservatism we aim to recover.
The strongest competitor on training-time safety is SAC-FDPI Breaking Safety Paradox. It is reported to achieve near-zero constraint violations across all 14 Safety-Gymnasium environments while matching or exceeding top baselines' reward. No scalar reward or cost values are reported there, and no reported result was found on Lacuna for cumulative training-time violations on these tasks. We will therefore rerun FDPI ourselves, counting violations generated by its dual policy as real violations, since on hardware they would be.
The strongest competitor on sample efficiency is ESPO. It reports 25–29% fewer samples and needed 5.7 million samples on SafetyReacher-v4 against 8 million for baselines. On reward near the boundary, the competitor is BSPG. It decomposes policy gradients into a component along the cost level set and a pull toward the budget, and is reported to beat CRPO and ESPO on SafetyPointGoal1 with cost residual near zero, but no scalar values are given.
All methods get the same environments, seeds (ten), environment-step budget, network sizes, and an equal hyperparameter search budget of 20 configurations each. Beating them means one of two outcomes. The first is at least a 2x reduction in cumulative training violations against SAC-FDPI (dual-policy violations included) at statistically indistinguishable final reward. The second is matching ESPO's sample count to reach the PPO-Lag final reward with fewer violations.
Three ablations isolate the key ingredient. An ESA-style variant dithers the learned cost critic instead of the plant, which tests whether plant measurement matters at all. A fixed-amplitude dither tests whether the boundary-aware envelope matters. A filter-off variant keeps only the near-boundary data and asks whether the replay effect alone helps.
Why not just…?
Why not just differentiate the learned cost critic and project actions onto its safe half-space? This is the cheapest competitor and is exactly what the ESA-style critic-dither ablation, plus an autograd version of it, measures. The gradient is free, but it is only as good as the critic's estimate near the boundary. FDPI's finding that cost critics degrade once violations stop is the evidence that this fails late in training. If the autograd projection matches our method on cumulative violations, the plant dither is unnecessary and we will say so.
Why not just learn a one-step dynamics model of the constraint signal and use its action gradient? This is the most serious threat. A local model of $g_{t+k}$ as a function of action could supply the same Jacobian without wiggling the actuators. We will train such a model on identical data and run the same filter with its Jacobian. Our expectation is that dither wins early in training and under contact, where models are poorly fit. The model may tie or win in smooth velocity tasks. If it wins everywhere, the contribution reduces to the near-boundary data collection, and the experiments will show that.
Why not just use random simultaneous perturbations instead of sinusoids? Random-sign perturbation estimates the same slope and handles many action dimensions without frequency planning. The argument for sinusoids is smoothness on hardware. Tactile insertion with extremum seeking found that CMA-ES's random sampling was too slow and produced jarring contact. That evidence is from black-box optimization, not perturbation-based slope estimation, so we include a matched random-sign dither ablation rather than assume the answer.
Why not just tighten the budget or add a large penalty? Safety-Gymnasium documents that penalties either get absorbed or produce frozen, "safe but useless" policies. A tightened budget trades reward for safety, which is exactly the gap BSPG targets. We include SAC-Lag with the budget shrunk to match our violation count and compare reward.
Proposed Experiments
Stage one uses the Safety-Gymnasium velocity tasks: Ant, HalfCheetah, Hopper, Walker2d and Humanoid, plus SafetyAntVelocity-v1. Here the constraint signal is directly measured speed, so the filter's assumptions hold cleanly. Stage two uses SafetyPointGoal1 and SafetyCarGoal1 with lidar distance as $g$; this tests lagged, position-type constraints. We also run Humanoid-Circle, where FOCOPS reports reward 1106.1 against CPO's 671.0 at cost limit 50, to check that the filter does not cost reward on a known reference task. Stage three, if stage one succeeds, transfers to tactile insertion on AutoMate-style pegs with peak strain as the constraint. There the reference point is the 98% success of pure extremum seeking wiggling, whose insertions often exceed 100 seconds. The question in stage three is whether the amortized policy keeps that success rate at much lower latency while never exceeding the strain limit.
The metrics are cumulative training violations, final normalized reward and cost, samples needed to reach a reward threshold, and wall-clock overhead. ESA reported up to 50% longer runtime per episode, so overhead is a real cost to track. We also measure late-training cost spikes and the critic's calibration error in a band within $2m$ of the boundary.
The decisive checks are three. First, the plant-dither variant must beat the critic-gradient and learned-dynamics projections on early-training violations. Second, late-training critic calibration must stay flat where SAC-Lag's degrades. Third, cumulative violations must fall at least 2x below SAC-FDPI's at matched reward. If the first check fails, the idea is not worth pursuing beyond its data-collection effect.
References
- Extremum-Seeking Action Selection for Accelerating Policy Optimization
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Large Gain Stability of Extremum Seeking and Higher-Order Averaging Theory
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Source Seeking in Unknown Environments with Convex Obstacles
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Extremum Seeking Controlled Wiggling for Tactile Insertion