A robot arm learning a contact task such as wiping, polishing, or insertion can easily hurt itself or its surroundings while it explores. Many recent systems let a reinforcement learning (RL) policy choose two things at each step: where the hand should go, and how springy the hand should be. The springiness is set through variable impedance control, meaning the arm behaves like a virtual spring and damper whose stiffness and damping the policy changes over time. Existing safety layers for these systems usually watch the measured contact force and step in when it gets too high. That makes safety only as good as the wrist force–torque sensor, which is noisy, drifts, and must be low-pass filtered. Filtering adds delay, and the policy itself runs 20 to 50 times slower than the arm's torque loop. Delayed, noisy measurements are the wrong basis for a hard safety guarantee.
We propose to certify safety from the commands instead of the measurements. An impedance-controlled arm can only push on the world as hard as its virtual spring is stretched, plus whatever momentum it carries into an impact. Both quantities can be computed from the commanded stiffness, the commanded target pose, and the joint encoders. The encoders are precise and effectively undelayed. We therefore place a small projection filter inside the 1 kHz torque loop. It keeps the policy's impedance command inside a "force budget" set: a coupled bound on spring stretch, stiffness, kinetic energy, and the energy injected by changing gains. The force sensor is used only for the task reward and the policy's observations, never for the safety decision. The claim is that this makes constraint violations during exploration largely insensitive to force-sensor noise and to policy-side latency, while staying less conservative than simply clipping gains.
Core Claim
For a Cartesian impedance controller, the force the end-effector applies to an obstacle it touches is approximately the spring term $K_t(x^d_t - x_t)$, where $K_t$ is the commanded stiffness, $x^d_t$ the commanded target pose, and $x_t$ the measured pose. On top of that comes an impact transient set by the arm's kinetic energy. If we assume an upper bound $\bar k_{env}$ on environment stiffness, converting the kinetic energy $E_t$ into a spring of that stiffness produces a peak force of at most $\sqrt{2\bar k_{env}E_t}$. This gives a sufficient condition that needs no force measurement:
$$\|K_t(x^d_t - x_t)\| + \sqrt{2\,\bar k_{env}\,E_t} \le F_{\max}, \qquad E_t = \tfrac12\,\dot x_t^\top \Lambda(q_t)\,\dot x_t .$$
Here $F_{\max}$ is the force limit, $\dot x_t$ is the end-effector velocity from the encoders, and $\Lambda(q_t)$ is the arm's reflected inertia at joint configuration $q_t$.
Raising stiffness while the spring is stretched injects energy $\tfrac12 e_t^\top (K_{t+1}-K_t)e_t$, where $e_t = x_t - x^d_t$. This is the passivity failure listed in the file's open problems for variable impedance control. We therefore add an energy reservoir, or "tank," $T_t$. Damping refills it, and gain increases must be paid from it:
$$\tfrac12\, e_t^\top (K_{t+1}-K_t)\, e_t \le T_t .$$
At each 1 kHz tick, the filter projects the most recent policy command onto the set defined by these two inequalities, together with box limits on gains. The first inequality is evaluated at the state predicted one tick ahead, so the check covers the inner loop's own delay. Two consequences follow. First, communication delay between the 20 Hz policy and the arm cannot break the guarantee, because the check is re-run locally against fresh encoder data. Second, force–torque noise enters nowhere in the certificate.
The set is coupled rather than a box, and that is the source of the claimed advantage over clipping. A clip safe against every combination must cap stiffness times offset at its worst case. The budget set instead lets the policy be very stiff when the spring is barely stretched, as in free-space tracking, or stretch the spring further when stiffness is low, as in gentle pressing. It forces slow approach speeds only when the assumed environment is stiff. The bound is conservative in known ways. It ignores the helpful effect of damping and assumes a spring-like contact. Whether it is tight enough to be useful is an empirical question the experiments are built to answer.
Relation to Prior Work
Four papers motivate or supply the method. VICES established pose-plus-gains as an RL action space, with a 20 Hz policy over a 500 Hz inner loop. We reuse that action space unchanged. Buchli et al. showed that learned impedance naturally stays soft with brief stiffness spikes. That pattern is exactly what our coupled set permits and a box clip would suppress.
The closest work is Safe OnGO-VIC. It treats gains as control inputs and runs a control barrier function (CBF) filter at 125 Hz; a CBF is a constraint that keeps the state inside a safe set by limiting how fast it can approach the boundary. Its barriers are workspace constraints evaluated on measured states. It has no RL explorer and no analysis of force-sensor corruption, and it was tested on simple surfaces. We differ on three counts: our constraint is a force and energy budget on the command itself, it sits under an exploring 6-DoF policy, and it is designed so that sensor noise is irrelevant to it.
CHEQ is the strongest learning-side competitor. It gates between an RL agent and a nominal controller using uncertainty from an ensemble of value estimators. It still produced 5 real violations, has no invariance guarantee, and smooths over 10 steps at 20 Hz. The contact-safe RL framework of Zhu et al. adds a 1 kHz momentum-observer reflex with heuristic mode switching. That reflex is itself a model-based estimate that can err. RMP-VICES shapes motion with hand-weighted geometric policies but cannot bound force.
Two papers constrain the design. Control Barrier Functionals shows that barriers depending on delayed states lose their guarantees. This argues against building the barrier on filtered force and in favor of our local, encoder-based check. Safe hierarchical navigation handles sensor noise by inflating margins with Kalman covariance. The file notes that in contact tasks, inflating force margins produces the "safe but useless" behavior identified by Li et al.. We avoid inflation by not measuring the constrained quantity at all.
Because the certificate is model-based, it also sidesteps the "safety paradox" of Feasible Dual Policy Iteration: there is no learned safety critic to be starved of near-boundary data.
Research Program
The first component is the filter. At each tick it solves a small convex program: the Euclidean projection of the commanded (stiffness, damping, target offset) onto the budget set, with gains parameterized per axis in the end-effector frame. The target is a closed-form or few-iteration solution well under 1 ms. The reference point for feasibility is the 0.68 ms safety filter reported by the navigation work. When the set is empty, which happens when the arm is already moving too fast, the filter falls back to lowering the offset and raising damping. We will prove that this fallback always restores feasibility.
The second component is learning. We train Soft Actor-Critic in the VICES action space. The critic sees the filtered action, and the policy pays a small penalty on the distance between its raw and projected actions, so it learns to stay inside the set rather than lean on the filter.
The third component is analysis. We will derive when the quasi-static-plus-impact bound upper-bounds true peak force for a mass–spring–damper contact of stiffness at most $\bar k_{env}$. We will then measure its slack empirically across environment stiffnesses, since that slack is the price of the guarantee.
Baselines
The naive reference is VICES with no safety layer, trained with SAC under the same reward. This shows how many violations exploration causes when nothing intervenes. The penalty baseline adds a force-violation term to the reward, with the weight tuned over the same grid budget as our method's single penalty weight. It is joined by PPO-Lagrangian. On Safety-Gymnasium velocity tasks, PPO-Lagrangian achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" (Safety Gymnasium). Both come from a different task family, but they indicate the trade-off to beat.
The strongest existing method is CHEQ, re-implemented on our polishing replica with its 20 Hz rate and 10-step uncertainty smoothing. Its reported numbers, on the bridge workpiece and not our tasks, are 56 failures in simulation versus over 6,000 for standard RL, and 5 violations over about 8 hours and 250,000 steps on a real Franka (CHEQ).
The second strong competitor is a measured-force barrier filter in the style of Safe OnGO-VIC. This is a CBF on low-pass-filtered F/T force, run at the same 1 kHz, with the filter cutoff tuned for each noise level. Its reported steady-state force variance, 0.09 to 0.14 N², on simple surfaces (Safe OnGO-VIC), serves as our target for smoothness.
The key ablation is box clipping. Its gain and offset limits are chosen so that it certifies the same $F_{\max}$ as our set in the worst case. This isolates whether coupling is what buys performance. A second ablation removes the energy tank, to show whether gain-injected energy matters in practice.
All methods share the reward, the observations (including F/T), the environment-step budget, and the number of seeds, set at five or more. Beating the field means two things at once. First, our method must show fewer ground-truth force violations than CHEQ and the penalty baselines during training, and a violation count that stays flat as sensor noise grows while the force-barrier baseline's count rises or its task success falls. Second, it must reach task success within the variance of unconstrained VICES and clearly above box clipping.
Why not just…?
Why not just clip stiffness and target offset to fixed limits? This is the most serious threat. If clipping at the worst-case-safe level matches our task performance, the coupled set adds nothing and the contribution collapses to "use VICES with limits." The prediction against it rests on Buchli et al.'s finding that good impedance profiles are soft with brief stiff spikes, and on CHEQ's report that a fixed stiffness setting "proved incapable of maintaining constant force across the varying slopes." Worst-case clipping removes exactly the combination of high stiffness and small stretch those findings call for. The box-clip ablation settles the question directly. If it ties, we will report that result.
Why not just filter the force signal and put a barrier on measured force? Filtering converts noise into delay, and Control Barrier Functionals shows that barriers on delayed states lose their invariance guarantees. That paper did not test manipulators, so the harm here is a hypothesis, not a known fact. Our noise and latency sweep measures it. If a well-tuned filtered-force barrier holds up to realistic F/T noise, our advantage narrows to latency robustness, and we will say so.
Why not just use CHEQ-style uncertainty gating? Gating reduces violations but gives no bound; CHEQ still recorded 5 hardware violations. Its safety also depends on the ensemble's uncertainty estimates staying calibrated under shift. Michelmore et al. found that lightweight uncertainty methods collapsed to a 0.00 safety score under unseen sensor noise. The head-to-head comparison under injected F/T corruption tests whether this failure mode transfers.
Why not just penalize force violations in the reward? Li et al. describe the dilemma this creates: small weights accept damage, large weights freeze the arm. The tuned-penalty and Lagrangian baselines quantify how many violations occur before convergence, which is the quantity that matters on hardware.
Proposed Experiments
In simulation, we use the robosuite wiping and door-opening tasks from VICES (robosuite) and a MuJoCo curved-surface polishing replica following CHEQ's setup. Two sweeps run on top of these tasks. The first injects F/T sensor corruption: white noise up to several newtons, slowly drifting bias, and quantization. The second injects policy-to-controller latency from 0 to 100 ms and inner-loop delay from 0 to 5 ms. We also vary environment stiffness across roughly three orders of magnitude to map how the bound's slack grows.
Violations are counted from simulator ground-truth contact force, never from the corrupted sensor. The metrics are cumulative violations during training, success rate, force-tracking error and steady-state force variance, filter compute time, and the fraction of ticks on which the filter intervenes.
On hardware, we use a Franka Panda for surface wiping and polishing on a foam, wood, and metal set. Violations are logged from an offline-filtered F/T trace, which is not available to the controller. Policy latency is induced artificially.
The decisive checks are three. The first is that ground-truth violations under our filter stay near zero and flat across the whole sensor-noise sweep, while the filtered-force barrier degrades or becomes over-conservative. The second is that the coupled set beats box clipping on success at equal certified $F_{\max}$, which is the check that could falsify the core claim. The third is that the guarantee survives up to 100 ms of policy latency with no change in violation rate. A null result on the second check would reduce the project to a noise-insensitivity argument for command-space limits, which we would still report.
Arm-link collisions and joint torque limits (the file's Gap 4) are out of scope, as are environments whose stiffness exceeds the assumed bound.
References
- Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
- Variable Impedance Control - A Reinforcement Learning Approach
- Safe Online Gain Optimization for Variable Impedance Control
- CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishing
- A Contact-Safe Reinforcement Learning Framework for Contact-Rich Robot Manipulation
- RMPs for Safe Impedance Control in Contact-Rich Manipulation
- Control Barrier Functionals: Safety-critical Control for Time Delay Systems
- Safe Hierarchical Navigation in Crowded Dynamic Uncertain Environments
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning