Explore to the Edge of Recovery: Base-Controller Safety Critics for Residual RL Under Sparse Rewards

Residual policy learning trains a neural network to add small corrections on top of a classical controller, such as a PID loop or a potential-field planner. It is one of the most reliable ways to make reinforcement learning (RL) sample-efficient on physical systems, with reported speedups of 5x to 10x over learning from scratch. It has no principled answer to two questions that matter on real hardware: how large may the learned correction be before it causes a collision or a force spike, and how does the agent keep exploring when the only reward is a binary "task done" signal? The standard safe-RL fix, penalizing constraint violations through a learned multiplier, interacts badly with sparse rewards. The penalty is dense and the reward is not, so the cheapest way to lower the objective's cost is to output a zero correction and never find the goal.

We propose to use the classical controller not only as a starting point but as a recovery policy. We train a critic that answers one question: "if I apply this action now and then hand control back to the base controller, will the system stay safe?" At every step, the learned correction is scaled down to the largest multiple that still passes this test. Safety is enforced by the size of the action, not by a reward penalty. The policy therefore optimizes the pure sparse success signal, and its exploration is pushed right up to the boundary of what the base controller can recover from.

Core Claim

The critic evaluates a fixed, known policy: take action $a$, then follow the base controller forever. That makes it much easier to learn than the feasibility functions used in constrained RL, which must track a policy that keeps changing. It also yields a simple invariance argument. Suppose the current state is recoverable under the base controller, and the executed action keeps the next state recoverable. Then, by induction, the system stays in the recoverable set, up to critic error. Our claim is that this construction, on tasks where the base controller alone succeeds only sometimes, does two things together. It reaches the sparse-reward success rates of unconstrained residual RL. It incurs training-time violations comparable to hard-switching safety priors, while avoiding the timidity of Lagrangian methods and the control discontinuities of switching.

Formally, let $u_b(s)$ be the base controller's action in state $s$, let $\pi_\theta(s)$ be the learned residual direction, and let $\underline{Q}_b(s,a)$ be a lower-confidence estimate from an ensemble of critics. It estimates the probability of no constraint violation over horizon $H$ when action $a$ is taken and $u_b$ is followed afterwards. The executed action is

$$u = u_b(s) + \alpha^*(s)\,\pi_\theta(s), \qquad \alpha^*(s) = \max\{\alpha \in [0,\alpha_{\max}] : \underline{Q}_b(s,\, u_b(s)+\alpha\,\pi_\theta(s)) \ge 1-\delta\},$$

where $\delta$ is the tolerated risk. The value $\alpha=0$ (pure base action) is always the fallback. The maximization is a one-dimensional line search over a small grid, so it adds negligible cost per step.

Relation to Prior Work

Three lines of work each supply one piece of this construction and leave the rest open.

The first shows that anchoring on a base controller cuts sample cost. Continuous Versatile Jumping Using Learned Action Residuals reports 10x data efficiency on a quadruped. RESPRECT reports a 5x speedup in grasping. Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies trains a PPO residual over a potential field using only sparse binary rewards, reaching 95.88% simulated success at 0.1mm tolerance, where the model-based controller alone falls to about 3%. That last paper is the closest motivating result: it shows residual RL can live on sparse rewards. It places no constraint at all on collisions or jamming during training. We keep its reward setting and add the safety mechanism it lacks.

The second line bounds residuals, but not with task-aware geometry. Adaptive control of a mechatronic system using constrained residual reinforcement learning confines the residual to a fixed relative tube around a PI controller, with a Lyapunov proof for 1-DOF tracking under dense rewards. Our scaling is state-dependent and driven by environment constraints, so the tube widens far from obstacles and narrows near them. DR-MPC blends MPC with a residual and falls back heuristically when an out-of-distribution detector fires. We replace that heuristic with a learned recoverability test.

The third line supplies the safety logic we reuse and alter. Safe Exploration via Policy Priors (SOOPER) switches wholesale to a safe prior when a pessimistic world model predicts budget violation, and reports zero violations in simulation and on a real racecar. Our backup reasoning is the same idea in model-free, continuous form. Instead of an all-or-nothing switch that truncates rollouts, the correction shrinks smoothly toward zero. Compliant Residual DAgger gives direct evidence that hard takeovers cause force discontinuities in contact tasks.

Two papers constrain the design and appear in our evaluation. Disturbance Observer-based Control Barrier Functions with Residual Model Learning applies residual learning to the dynamics model and filters an end-to-end policy through a control barrier function (a hand-designed function whose sign marks safe versus unsafe states, enforced by a quadratic program). It is the natural "filter instead" competitor.

Two findings motivate the reward-free safety channel. Safe Policy Exploration Improvement via Subgoals reports SAC-Lagrangian at 0% success in narrow corridors. Quanyi Li et al. documents that reward penalties yield either reckless or "safe but useless" agents.

Breaking Safety Paradox with Feasible Dual Policy Iteration warns that as a policy becomes safe, its data stops containing boundary examples. Our critic is partly exposed to this problem, and we address it directly below.

Research Program

We will build the method on an off-policy actor-critic, soft actor-critic with a residual action head. The replay buffer stores the executed scaled action, so the reward critic learns from what actually happened. The recoverability ensemble is trained by fitted policy evaluation. Its bootstrap target at the next state uses the base controller's action, which is known and deterministic, so there is no maximization and no dependence on the learner's current policy.

To counter the safety paradox, in simulation we periodically launch short probe rollouts of the base controller from buffered states, perturbed by random residuals of up to $\alpha_{\max}$. These cheaply label near-boundary states, and every step they spend is counted in the violation budget. On hardware, probes are disabled, and the ensemble's lower bound together with a conservative $\delta$ carries the risk.

We will also run the analysis the method invites. We will measure how often the induction premise fails, meaning the method entered a state the true base controller cannot recover from. We will relate that rate to ensemble calibration error, since the invariance argument predicts that violations should track the critic's false-positive rate.

The regime variable we treat as central is base-controller competence. Compliant Residual DAgger reports that residual methods need a base that already succeeds roughly 10–20% of the time. We will sweep base competence by degrading the potential-field gains, and report where the method stops working rather than hiding it.

The benchmarks are two. The first is sparse-reward variants of Safety-Gymnasium PointGoal and CarGoal (Safety Gymnasium), with a scripted potential-field base controller and the native hazard cost. The second is the IsaacGym tight-insertion setup of Integrating Model-based Control and RL, with its potential-field base and sparse binary reward, plus a cost for contact force above a fixed threshold and for jamming.

Baselines

The naive reference points are two. One is the base controller alone. The other is unconstrained additive residual RL on the same sparse reward, which reproduces Integrating Model-based Control and RL and gives the upper envelope for success and the lower envelope for safety. For insertion, that paper reports 95.88% simulated success at 0.1mm tolerance under high noise, versus about 3% for model-based control. Our method must come close to that success while cutting training violations substantially.

The first strong competitor is residual RL trained with a Lagrangian cost multiplier, using the PID-damped multiplier update that Safety Gymnasium shows to be the most stable (CPPO-PID style). This is the obvious "just add constrained RL to residuals" design and the direct test of the timidity hypothesis. That paper reports PPO-Lag cutting violations by 98% relative to PPO at a 45% reward cost on velocity tasks. No reported result was found on Lacuna for Lagrangian residual RL under sparse rewards. We beat this baseline if we achieve higher final success at equal or lower cumulative training violations.

The second is a SOOPER-style hard switch (Safe Exploration via Policy Priors). It uses the same recoverability ensemble and the same base controller, but the full base action takes over whenever the test fails, rather than the residual being scaled. This isolates continuous scaling from switching. SOOPER reports zero simulated violations and a hardware policy twice as good as its prior. No reported result was found on Lacuna for it under sparse rewards or contact tasks. We beat it by matching its violation count while achieving higher success or lower peak contact force and jerk.

The third is a filter competitor: residual RL whose total action passes through a hand-designed control-barrier-function QP, in the spirit of RES-DOB-CBF. That paper reports 146 to 2.07 collisions per trip on a physical racecar, a different setup, so the figure is context only.

The fourth is a fixed-tube ablation following constrained residual RL, with a constant $\alpha$ tuned per task.

The key-ingredient ablation estimates the recoverability critic for the current learned policy instead of the fixed base controller. If our argument is right, this version should show noisier estimates and more violations.

For fairness, all methods share the same base controller, network sizes, environment-step budget (probe rollouts counted as environment steps), five seeds per task minimum, and an equal hyperparameter search budget of 20 configurations each.

Why not just…?

Why not just use a control barrier function filter on the combined action? This is a real threat on the navigation tasks. Hazards there have simple geometry, so a barrier function is easy to write. It is less natural for insertion, where the unsafe set is "jamming or excessive contact force" and depends on contact dynamics that no simple analytical barrier captures. That is exactly where Integrating Model-based Control and RL shows the analytical model fails. Our experiments settle the question per domain. If the filter matches us on Safety-Gymnasium, we will say so, and restrict our claim to domains where the unsafe set must be learned.

Why not just use a fixed relative tube around the base action? A constant tube must be tight enough for the most dangerous state. It is therefore overly tight in open regions, which slows sparse-reward discovery, and possibly too loose at boundaries. The tuned fixed-tube baseline directly measures this cost. If a single tuned $\alpha$ achieves the same success-violation tradeoff, the learned critic is not worth its complexity and we will report that.

Why not just add a Lagrangian penalty to residual RL? Under sparse rewards the penalty gradient is the only dense signal, which pushes the residual toward zero. Safe Policy Exploration Improvement via Subgoals reports 0% success for SAC-Lagrangian in long corridors. The residual-Lagrangian baseline tests whether the base controller's own progress is enough to rescue Lagrangian methods. If it is, our central motivation weakens.

Why not just switch to the prior, as SOOPER does? Switching is proven safe. In contact tasks, however, it causes force discontinuities (Compliant Residual DAgger), and truncated rollouts hurt credit assignment. The matched-critic switching baseline separates these effects from the critic itself.

Proposed Experiments and Decisive Checks

We report four metrics throughout: sparse success rate against environment steps, cumulative training violations including probe steps, expected maximum consecutive cost steps from Revisiting Safe Exploration to capture how severe violations are, and, for insertion, peak contact force and action jerk.

Four checks decide the proposal.

First, at base competence around 20%, our method should reach at least 90% of unconstrained residual RL's final success while incurring at most one tenth of its training violations. Residual-Lagrangian should show lower success at comparable violations.

Second, the competence sweep should reveal a threshold below which all residual methods fail. That threshold would bound the claim honestly.

Third, the fixed-base critic should show lower calibration error and fewer premise failures than the current-policy critic. This would confirm that the "evaluate a fixed policy" design is what makes it work.

Fourth, versus switching, continuous scaling should give lower peak force and jerk in insertion at matched violations.

If the tuned fixed tube or the barrier-function filter matches our method on both domains, we will conclude that learned recoverability adds nothing over simpler geometric bounds.