Residual Hierarchies That Widen Their Own Leash

Long robot tasks such as opening a microwave, then sliding a cabinet door, then turning a knob break flat reinforcement learning (RL) because small errors compound and rewards arrive late. Hierarchies help: a slow "manager" proposes intermediate targets (subgoals), and a fast "worker" steers toward each one. They are hard to train, though. When manager and worker learn at the same time, each keeps changing the other's problem. A separate line of work, residual policy learning, avoids learning from scratch altogether. A learned network adds a small correction on top of an existing controller that already mostly works. We propose putting a residual at both levels. The manager corrects a coarse planner's subgoals, and the worker corrects a base controller's actions. The key design choice is that the manager's corrections are allowed to grow only as fast as the worker has actually learned to reach beyond what the base controller could reach.

The underlying bet is about stability. When the worker is "base controller plus bounded correction," most of its behavior is fixed. The manager's view of the world should then drift much less than in a hierarchy learned from scratch. If that is true, the hard problem of training both levels together becomes a controllable one. We can measure and exploit how much the worker's reachable region grows, instead of fighting it with sensitive adversarial penalties or frozen offline datasets.

Core Claim

Existing systems cover this space only in pieces. HiWET combines a hierarchy with a low-level residual over a pretrained kinematic prior for humanoid loco-manipulation. Its high-level policy, however, outputs unconstrained absolute commands and is not co-adapted with the residual tracker. Joint-training methods constrain a from-scratch manager to what a from-scratch worker can reach:

  • CRISP re-parses expert demonstrations against the current worker and adds an imitation-style discriminator.
  • DIPPER puts the worker's value function inside a preference-optimization objective trained on a fixed offline dataset.
  • SAGA penalizes subgoals that a learned discriminator judges unreachable.

None of the three uses a base controller or base planner. Residual methods such as Residual Policy Learning, ResFiT and DAWN are all flat. DAWN explicitly leaves hierarchical, long-horizon settings open.

Our claim has two parts. The first is that a two-level residual hierarchy has a manager-side non-stationarity that is small, measurable, and driven only by the worker's residual. The second is that tying the manager's correction budget to measured growth in the worker's reach gives stable joint training with no hand-tuned feasibility weight.

Concretely, a base planner supplies $g_{\text{base}}(s)$, for example waypoints from a motion planner or from stage boundaries of a single demonstration. The manager outputs a correction $\Delta g$, and a base controller moves toward the resulting subgoal:

$$g = g_{\text{base}}(s) + \Delta g, \qquad a = a_{\text{base}}(s, g) + \Delta a, \quad \|\Delta a\| \le \beta_L .$$

Here $s$ is the state, $a$ the executed action, and $\beta_L$ a fixed bound on the worker's correction. A feasibility critic $V_L(s,g)$ estimates the probability that the worker reaches $g$ within the manager's time window. The manager maximizes task return subject to a reachability constraint:

$$\max_{\pi_H} \; J_H(\pi_H) \quad \text{s.t.} \quad \mathbb{E}\big[V_L(s, g_{\text{base}}+\Delta g)\big] \ge \tau .$$

The threshold $\tau$ is fixed, and the constraint is enforced with a learned Lagrange multiplier. The effective leash on $\Delta g$ therefore widens only where $V_L$ has learned that the worker's residual extends reach. At initialization $V_L$ reflects only the base controller's reach, so the manager starts close to the base planner.

Research Program

The first component is the feasibility critic. It is where the two literatures collide. DAWN shows that residual critics suffer from two problems. The first is a cold start: a random critic sends destructive gradients to an already competent policy. The second is a scale mismatch: small corrections are invisible next to large base actions. DAWN fixes both with a warmup buffer of pure base-policy data plus layer normalization. In a hierarchy the stakes are higher, because a blind worker critic hands the manager a corrupted feasibility signal. We will anchor $V_L$ on base-controller rollouts toward base-planner subgoals, so at time zero it is a stationary estimate of the base controller's reach. Two further choices follow DAWN: $V_L$ uses layer normalization, and it takes $\Delta a$ as an input feature on a normalized scale.

We will also settle, for the hierarchical case, the conflict the file identifies between gradient-based critic burn-in, as in Residual Policy Learning, and DAWN's passive warmup. We apply each warmup scheme at each level separately. This matters because the manager's policy is also near-deterministic at start, so DAWN's warning about the entropy term growing without bound may apply there too.

The second component is an analysis of non-stationarity. We define the transition invalidation rate as the fraction of stored manager transitions whose outcome state changes by more than a tolerance when replayed under the current worker. Here a stored manager transition means a start state, subgoal, and reached state. We will track this rate across training for three worker types: residual, from-scratch, and frozen. We will also derive a simple bound. If $a_{\text{base}}$ is fixed and $\|\Delta a\| \le \beta_L$, the reached state after $c$ steps can shift by at most roughly $c\,\beta_L L$ between worker iterates, where $L$ is a Lipschitz constant of the dynamics. The drift the manager sees is thus governed by $\beta_L$ rather than by the worker's whole behavior. The empirical question is whether this loose bound predicts the measured ordering, and where contact dynamics break it.

The third component tests robustness to base-planner quality, which is the regime where hierarchy should pay off over flat residuals. We will corrupt $g_{\text{base}}$ in a controlled way. The corruptions are Gaussian offsets, systematically misplaced waypoints (for example, a subgoal on the wrong side of an obstacle), and missing stages. Each is swept from mild to severe.

Baselines

The naive reference point runs the base planner and base controller with no learning. It shows how much of any success comes from the priors. The second reference is flat RL from scratch. Residual Policy Learning reports this reaches 0% on the sparse-reward Hook tasks, so we expect it only as a floor.

The strongest flat competitor is a single residual policy over the full composed system of base planner and base controller. We train it with the ResFiT off-policy recipe of high update-to-data ratio, symmetric demo/online sampling and multi-step returns, and add DAWN's warmup and layer normalization. This baseline receives exactly the same priors as our method and the best known residual training tricks. It is the most direct test of whether the manager earns its keep. On the MuJoCo Hook tasks the reported number for flat residual learning is roughly 80% success (Residual Policy Learning). ResFiT's reported numbers are real-hardware results on different tasks and are not directly comparable. On ManiSkill, DAWN reports reaching 90% on PegInsertionSide about 6x faster than prior methods (DAWN). That is a short-horizon task that we use only to check our worker implementation.

The strongest hierarchical competitors are CRISP, DIPPER and SAGA. We run them with their published structure and give them the same single demonstration, where they use one, and the same environment-step budget. On Ant Push, SAGA reports 0.72 success versus 0.62 for LESSON (SAGA). On Franka Kitchen, CRISP and DIPPER report only relative gains of more than 40% and up to 40% over baselines. No reported absolute Franka Kitchen success rate for either was found on Lacuna, so we will rerun both ourselves. To separate the contribution of priors from that of the coupling rule, we also run CRISP and SAGA with our residual worker substituted in.

Four ablations isolate the key ingredient. The first replaces the Lagrangian reachability constraint with a fixed-weight penalty swept over the range SAGA reports as sensitive. The second drops the high-level residual, so the manager outputs absolute subgoals over a residual worker, which is essentially a jointly trained HiWET-style system. The third keeps a residual only at the high level, over a from-scratch worker. The fourth trains in stages: the residual worker is trained and frozen, then the residual manager is trained.

All methods get identical environment-step budgets, five seeds or more, and the same hyperparameter-search budget per method. Beating a baseline means higher final success with non-overlapping 95% confidence intervals across seeds, or matching success in at most half the environment steps.

Why not just…?

Why not just use a flat residual policy over a scripted multi-stage base? This is the real threat. Residual Policy Learning already solves long, sparse Hook tasks this way. When the base plan is roughly right, a flat residual may match us. Our argument is that a bounded action correction can only fix local errors. When a waypoint is on the wrong side of an obstacle, the correction needed exceeds $\beta_L$, and widening $\beta_L$ destroys the prior's value, the tension noted in section 4 of the file. The planner-corruption sweep settles this. We predict the two methods tie at low corruption and our method wins once waypoint errors exceed the worker's reach. If the flat residual keeps pace across the sweep, the high-level residual is unnecessary and we will report that.

Why not just freeze the residual worker and then train the manager? Staging removes non-stationarity entirely. Zhang et al. argue that frozen lower levels cap performance. DAC and Option-Critic show gains from concurrent training, but those results come without residual priors. Our non-stationarity analysis may show that residual workers drift so little that staging costs nothing. That would be a useful negative result. The staged ablation measures directly whether co-adaptation adds success once the worker is residual.

Why not just plug a residual worker into CRISP or DIPPER? This is plausible and is included as a baseline. CRISP needs resets to arbitrary intermediate demonstration states. DIPPER's high level is tied to a fixed preference dataset and cannot move toward subgoals the data never contained. Our coupling needs neither, but it has to show it is at least as good as these hybrids to justify itself.

Why not just fine-tune a pretrained policy end to end? Full fine-tuning erodes the prior's global competence (Pari et al.), which is the reason residual methods exist. We include a full-fine-tuning worker only as a secondary check.

Proposed Experiments and Decisive Checks

We evaluate on four families of tasks, each chosen to supply a comparison the others cannot:

  • MuJoCo Ant Push, where a coarse grid planner supplies base subgoals and a reported SAGA number anchors the comparison.
  • Franka Kitchen, where stage boundaries from the single demonstration used by DIPPER give base subgoals and an operational-space controller gives base actions.
  • The Silver et al. Hook tasks, the home ground of the strongest flat residual baseline.
  • Several multi-stage RLBench tasks, whose built-in waypoint planner is a natural base planner. RLBench has no reported learning results on Lacuna, so all numbers there will be our own.

We report five metrics: success rate over environment steps, transition invalidation rate, the fraction of proposed subgoals actually reached, the magnitude of $\Delta g$ and $\Delta a$ over training, and calibration of $V_L$ against empirical reach.

Three outcomes would decide the proposal:

  1. Drift check. Invalidation rate should be lower for the residual worker than for the from-scratch worker by a clear margin, and its ordering across values of $\beta_L$ should follow the drift bound. If it does not, the stability premise fails.
  2. Planner-corruption check. Across the corruption sweep, the dual-residual hierarchy should beat the flat residual baseline once waypoint errors exceed the worker's reach, and not before. A curve showing where that crossover happens is the central figure.
  3. Coupling check. Replacing the Lagrangian constraint with a fixed penalty should reproduce SAGA-style sensitivity, with success varying strongly across weights, while the coupled version stays within seed noise across thresholds $\tau$. If the fixed penalty is equally robust once priors are present, the coupling adds nothing beyond the priors themselves, and we will say so.