Residual reinforcement learning trains a small neural "correction" on top of an existing controller that already mostly works, such as a PI loop, a motion planner, or a pretrained policy. The learner then adjusts that controller's output instead of discovering behavior from scratch. It is one of the most sample-efficient ways to put RL on real robots. It rests on one fixed design choice: how far the correction is allowed to push the base controller. Set that bound too tight and the correction cannot fix the base controller where it is wrong. Set it too loose and exploration noise drives the system into unsafe states. Today this bound is a hand-tuned constant. We propose to make it a decision. A slow, high-level policy chooses, every few hundred milliseconds, how much authority the fast residual correction gets. It is trained under an explicit budget on cumulative safety cost during training.
The two-sentence version is this. A trusted base controller becomes a "safe fallback" action (zero authority) whose cost is known in advance, and a high-level policy learns where and when it is worth leaving that fallback. Safe exploration then reduces to a one-dimensional, temporally committed choice. It stays cheap to enforce at every decision, unlike safety filters over the full action space or planning in a learned model.
Core Claim
Let $a^{\text{base}}(s)$ be the base controller's action in state $s$. At the start of each window of $K$ control steps, the high-level policy $\mu$ picks an authority level $\alpha \in [0, \alpha_{\max}]$. The low-level residual $\pi_\theta$ is conditioned on $\alpha$ and acts at every step:
$$a_t = a^{\text{base}}(s_t)\,\big(1 + \alpha\, \tanh \pi_\theta(s_t, \alpha)\big).$$
The multiplicative form comes from constrained residual RL (CRRL). That work scales a Soft Actor-Critic residual relative to an industrial PI controller with a fixed bound $\beta_r$. It proves bounded tracking error during exploration and reports a 13% error reduction on a slider-crank rig. We keep its bounded form and turn the constant into a learned, state-dependent, temporally extended choice.
The high level is trained on a constrained Markov decision process. In that setting the agent maximizes return subject to expected discounted cost being at most a budget $\epsilon$. It selects $\alpha$ with a state-wise feasibility test borrowed from backward value functions. That paper learns a cost-so-far estimate alongside a cost-to-go estimate. It shows that requiring their sum to stay below the budget keeps a policy feasible throughout training, provided it starts from a safe policy and takes bounded steps. In our setting the high level picks
$$\alpha_k = \max\{\alpha : C_{\text{past}}(s) + Q^H_c(s,\alpha) \le \epsilon\},$$
tie-broken by the reward critic. Here $C_{\text{past}}$ is cost accrued so far in the episode and $Q^H_c(s,\alpha)$ is the high-level cost critic over the next window and beyond. Because the base controller is assumed safe on its own, $\alpha=0$ is always available and its cost is anchored by data from the base controller. The feasible set is never empty. The search is over a scalar, so enforcing it costs one pass over a small grid rather than thousands of imagined rollouts.
What would be new is the combination of three things. Authority over the residual becomes the high-level action. The hierarchy's temporal commitment is what makes the cost critic learnable. And the known-safe zero-authority action anchors that critic, so that pure residual-plus-constraint methods and hierarchical subgoal methods can each be tested against it.
Relation to Prior Work
The direct motivation is an open problem in residual RL. When the base controller and the correction pursue conflicting objectives, a static sum $a^{\text{base}}+\Delta a$ produces destructive interference and chatter. When the base is badly biased, the residual wastes effort cancelling it. Both are documented in Residual Reinforcement Learning for Robot Control and Learning Vision-based Reactive Policies for Obstacle Avoidance. The first learned Sawyer block assembly in under three hours and kept 75% success on misaligned blocks where the hand-designed controller failed 90% of the time. The second paired RMPflow with a TD3 residual on β-VAE image features and reached 84% versus 39% for learning from scratch. Neither treats safety as a constraint during training. Both fix the residual's authority.
Three safe-exploration systems use a prior in a way close to ours, and each differs in a specific respect. SOOPER hands control entirely to a safe prior when a world model predicts budget violation or high uncertainty. It reports zero training collisions and twice the prior's reward on a physical race car. That is binary switching that depends on a calibrated world model, whereas our authority is continuous and model-free. HiSaRL guards a from-scratch low-level policy with a neural Lyapunov shield under an A/RRT planner. Its hard override can repeatedly fire and stall the agent when subgoals are infeasible. SPEIS uses high-level subgoals and a KL nudge to stop a SAC-Lagrangian worker from freezing, reaching 84% success in POLAMP corridors where SAC-Lagrangian reached 0%. Its worker has no base controller to fall back on. In all three, the hierarchy decides where to go. In ours, it decides how much to trust the learner.
Other work shapes the design without supplying the mechanism. DeCOM decomposes actions into a task part and a constraint-reducing perturbation, but at a single time scale. Budgeted Hierarchical RL shows that a high level paying for a costly resource learns to spend it at topological bottlenecks such as doorways. That motivates our prediction that learned authority will concentrate near obstacles and contact events. Feasible Dual Policy Iteration constrains the design. It shows that cost critics decay as violation data disappears from the replay buffer. Our anchored zero-authority action is meant to prevent this failure, and we test whether it does.
Research Program
We will build the two-level learner on top of SAC. The low-level residual is conditioned on $\alpha$ and trained on all transitions. The high-level policy acts every $K$ steps and carries two critics: a reward critic and a cost critic $Q^H_c(s,\alpha)$. They are trained on window-level returns, with the budget enforced by the feasibility test above plus a PID-tuned Lagrange multiplier as a soft backstop. The multiplier follows the CPPO-PID idea that Safety Gymnasium found damps oscillation.
Because the residual's behavior at a given $\alpha$ changes as it learns, the high-level critics face the non-stationarity familiar from hierarchical RL. We address it by relabelling window returns with the current residual only for $\alpha$ values near those taken. We treat this as an empirical question rather than claiming it is solved.
The analysis has three threads. First, we map where learned authority goes: near obstacles, at contact, or where the base controller is biased. We also measure whether it follows predictable structure. Second, we compare how accurate the cost critic is at the end of training against a flat critic, to test the anchoring argument against the safety-paradox decay reported by FDPI. Third, we deliberately corrupt the base controller with gain bias and an offset target, to measure how gracefully the method raises authority when the prior is wrong.
Baselines
The naive reference point is the base controller alone. Every other method must beat its return, and none should exceed its cost by much during training. Next is fixed-bound residual RL in the style of CRRL. This is the same residual network and multiplicative form with a constant $\alpha$, swept over the grid our high level can choose from, with the best constant chosen per task in hindsight. It is also run with a linear annealing schedule. Beating this oracle-tuned constant on training cumulative cost at equal final return is the central test. It is the most direct competitor, since it is exactly our method with the decision removed.
The strongest existing safe-exploration competitor with a prior is SOOPER. It is given the same base controller as its safe prior. Beating it means matching its reported zero-violation training regime while reaching higher return without a world model, or showing fewer interventions. The strongest hierarchical safe baseline is SPEIS. On POLAMP Level 2 corridors it reports 84% success and about 3% collisions, against 0% success for SAC-Lagrangian. Beating it there means higher success at equal or lower collision rate. We also run flat SAC-Lagrangian with the residual but no authority control. On the Franka vision obstacle-avoidance task, the reference is the residual method of Aljalbout et al. at 84% single-obstacle success and 72% on unseen cylinders. For Safety-Gymnasium Goal tasks run with a scripted base controller, I found no reported result on Lacuna for residual methods. There we report our own baselines only.
The ablation that isolates the key ingredient is a flat gate. One policy outputs both the residual and $\alpha$ at every control step, under the same cost critic and feasibility test. The only differences are the absence of temporal commitment and the per-step credit assignment. All methods share network sizes, the same base controllers, the same environment-step budgets, and an equal hyperparameter-search budget of trials per method, reported over at least five seeds.
Why not just…?
Why not just tune or anneal the residual bound? If authority needs vary across the state space, a single constant must be as small as the most dangerous region requires. That caps improvement everywhere else. The CRRL results show a fixed bound is safe but say nothing about what it costs in return. We test this directly with the hindsight-best constant and a schedule. If either matches us on the return-versus-training-cost curve, we will conclude that allocating authority is unnecessary on these tasks, which is itself a useful negative result.
Why not just learn a state-dependent gate at every step, without a hierarchy? This is the real threat. It could plausibly work as well. Our argument for temporal commitment is twofold. A window-level cost critic sees returns that aggregate $K$ steps and have lower variance. And committed authority prevents the fast switching that the literature associates with chatter. Neither argument is proven. The flat-gate ablation settles it on training cost, final return, and action jerk. If the flat gate matches, we will report that authority should be state-dependent but not temporally abstracted, and we will drop the hierarchical claim.
Why not just put a safety filter or shield on an unconstrained residual? Filters need a dynamics model. RES-DOB-CBF shows that barrier filters break under model error and needed a learned residual dynamics model plus a disturbance observer to hold up under 50% parameter error. HiSaRL's hard override can stall the agent. Filters also give the learner no signal about where exploration is worth risking. We include a shielded residual baseline in the POLAMP and Safety-Gymnasium runs, and we count intervention frequency alongside cost.
Why not just use SOOPER's switching? Switching is the special case where $\alpha \in \{0,\alpha_{\max}\}$ is chosen by a model. Our method restricted to that binary set is a direct model-free analogue. We run it to see whether continuous authority actually matters.
Proposed Experiments
The first testbed is Safety-Gymnasium Point-Goal and Car-Goal, levels 1 and 2. The base controller steers straight toward the goal and slows down with a potential field near hazards, so it is safe but slow. The second is POLAMP narrow corridors with a conservative lane-following base controller. The third is the Gazebo Franka obstacle-avoidance task with an RMPflow base, where residual numbers already exist. The bias-injection study reruns all three with the base controller's gains or target deliberately corrupted.
The primary metrics are cumulative cost over the whole training run, final return or success rate, and the fraction of episodes exceeding the budget during training. Secondary metrics are action jerk, how accurately the cost critic predicts held-out boundary states late in training, and maps of learned $\alpha$ across the state space.
The decisive checks are three. The method should dominate the hindsight-tuned fixed bound on the curve of training cost against return. It should beat or match the flat gate on that curve with lower jerk. Otherwise the hierarchy is not justified. And under a biased base controller it should raise authority where the bias bites and keep training cost below SPEIS and SAC-Lagrangian, rather than failing along with its prior.
References
- Adaptive Control of a Mechatronic System Using Constrained Residual Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Residual Reinforcement Learning for Robot Control
- Learning Vision-based Reactive Policies for Obstacle Avoidance
- Safe Exploration via Policy Priors
- HiSaRL: A Hierarchical Framework for Safe Reinforcement Learning
- Safe Policy Exploration Improvement via Subgoals
- DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
- Budgeted Hierarchical Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning