A hierarchical reinforcement learning agent splits control between two policies. A slow "manager" picks intermediate subgoals, such as where the gripper should be in one second. A fast "worker" issues motor commands to reach each subgoal. The best recent methods keep the manager from asking for impossible subgoals by consulting the worker's value function, which is the worker's own learned estimate of how well it will do from here. If that estimate is low, the subgoal is treated as infeasible. This works in clean simulation. On a real robot it has a specific weakness. The worker's value is computed from all of its sensors at once, so a camera glitch or a spurious depth reading lowers the value exactly as a real joint limit would. The manager then abandons a perfectly reachable subgoal because of a perception error it cannot tell apart from a motor limitation.
We propose to split the worker's reward, and therefore its value, into channels that follow the robot's sensors. The first is a kinematic channel covering reach progress, joint limits, torque and balance. Its critic sees only proprioception (joint encoders and IMU), which is nearly noise-free across the sim-to-real gap. The second is an exteroceptive channel covering obstacle clearance. Its critic sees camera, depth or LiDAR. The manager treats the kinematic value as a hard feasibility check and weights the exteroceptive value by how much the current perception can be trusted. In two sentences: sensor noise hurts hierarchies most at the manager–worker interface, where perceptual failures masquerade as motor infeasibility. Routing reward channels to the sensors that actually determine them keeps the feasibility signal clean without making the policy blind.
Core Claim
Under exteroceptive corruption, a hierarchical agent whose manager is regularized by a sensor-routed, decomposed worker value will reject far fewer truly feasible subgoals than one regularized by a scalar worker value. It will also complete more long-horizon tasks, and it will not pay for this with more collisions. The prediction is falsifiable in simulation and checkable on hardware. It concerns a quantity no prior work measures: the false-infeasibility rate, meaning the fraction of subgoals the manager discards or down-ranks even though an oracle with clean state judges them reachable and collision-free.
Where This Sits
The method we modify is DIPPER, described in Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction and Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach (also indexed as art_e4f4f88b; Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach). DIPPER trains the manager with direct preference optimization, which learns from pairwise comparisons of trajectories, and adds a penalty whenever the worker's scalar value $V^{\pi L}(s,g)$ says a subgoal $g$ is unreachable. It reports up to 35–40% success gains over HAC, SAGA and PIPER, but only on noise-free MuJoCo states. We keep its bilevel structure and replace the single scalar value with routed channels.
Reward decomposition, meaning training separate critics for separate reward terms, supplies the mechanism. Multi-Critic Actor Learning shows that separate critics avoid gradient interference. It also shows that fully separate networks beat a shared trunk with multiple output heads, which constrains our design toward disjoint critics. Humanoid Whole-Body Locomotion on Narrow Terrain motivates the proposal directly. Giving balance its own critic took a blind humanoid from 0% to 90% on a 25 cm path, but that policy is flat and uses only proprioception. Hierarchical goals contextualize local reward decomposition explanations already combines a hierarchy with decomposed Q-values, but only for explaining behavior in a noise-free 2D arena. It never feeds the decomposed values back into subgoal selection and never touches sensor noise.
In every prior multi-critic system all critics receive the same observation. When that observation is corrupted, all critics fail together. Routing each critic to its own sensors is the new ingredient.
For sensor robustness, METRICMM (Geometry of Uncertainty) provides our trust signal and our corruption protocol. It embeds sensors in a latent space where distance tracks transition cost and down-weights sensors whose readings disagree with a learned dynamics prediction. It is flat and scalar, so we reuse its disagreement score as the confidence on the exteroceptive channel rather than as a replacement for decomposition. Monolithic vs. hybrid controller for multi-objective Sim-to-Real learning is the closest physical evidence. On a Franka Panda with RealSense and AprilTag tracking, splitting reaching from avoidance into two hand-switched policies raised success from 30% to 90%. It used a hard distance threshold rather than a learned manager, and the authors report chattering at the switching boundary. Hierarchical RL for soccer shooting shows a 1 Hz / 30 Hz hierarchy transferring to hardware, but with a scalar reward at each level and simplified ball tracking.
Mechanism
The worker's reward is split as $r = r_{\text{kin}} + r_{\text{ext}}$.
- $r_{\text{kin}}$ covers progress toward $g$ plus joint-limit and effort penalties.
- $r_{\text{ext}}$ covers clearance from obstacles.
Two disjoint critics are trained on these channels:
- $V_{\text{kin}}(p, g)$ sees only proprioception $p$ and the subgoal. Because it cannot see obstacles, it estimates "reachable if unobstructed." We argue this is the right meaning for kinematic feasibility, and we test whether the missing obstacle information matters.
- $V_{\text{ext}}(\tilde{o}, p, g)$ additionally sees the possibly corrupted exteroceptive observation $\tilde{o}$.
The manager's DIPPER penalty becomes
$$\mathcal{R}(g) = \lambda_{\text{kin}}\,[\tau - V_{\text{kin}}(p,g)]_+ \;+\; \lambda_{\text{ext}}\,c_t\,[\tau' - V_{\text{ext}}(\tilde{o},p,g)]_+ ,$$
where:
- $[x]_+ = \max(x,0)$;
- $\tau$ and $\tau'$ are feasibility thresholds;
- $\lambda_{\text{kin}}$ and $\lambda_{\text{ext}}$ are Lagrange weights;
- $c_t \in [0,1]$ is the perception confidence at time $t$, computed from METRICMM's disagreement between observed and predicted latent state.
This penalty is used in two places. It regularizes the manager during training, as in DIPPER. It is also used at deployment as a runtime filter over candidate subgoals, because on hardware the manager must judge feasibility from real, noisy observations.
The design has a known risk. When $c_t$ is low because a real obstacle looks anomalous, down-weighting $V_{\text{ext}}$ could let the manager drive into it. Collision rate is therefore a primary metric, not an afterthought. When confidence is low, the manager's fallback is to choose shorter subgoals, not to ignore clearance.
Research Program
The work has three stages.
The first is diagnostic. We train DIPPER on its own benchmarks, apply METRICMM's seven corruption families to the exteroceptive channels only at test time, and measure how often the scalar-value penalty rejects subgoals that are feasible under the clean state. If false rejections are rare, the premise is wrong and we stop.
The second stage builds the routed decomposition and compares it against the baselines below. The benchmarks are DIPPER's Maze Navigation, Pick and Place, Push and Franka Kitchen, augmented with depth or RGB obstacle observations. We also use Fetch–Slide and Fetch–PickAndPlace in METRICMM's multimodal setup.
The third stage is a hardware check. It replicates the obstacle-avoiding reach task from the monolithic-vs-hybrid study on a Franka Panda with the same camera pipeline, trained in simulation with the same randomization budget for every method.
Baselines
The naive reference point is DIPPER as published: a scalar worker value and no noise handling, trained on clean states and tested under corruption. This measures how much damage corruption does to an unprotected hierarchy.
The strongest existing competitor on the robustness question is DIPPER with a METRICMM front end. METRICMM fuses and filters observations before a scalar worker critic, so a cleaned state feeds the ordinary feasibility penalty. On Fetch–Slide under sensor failure at frequency 0.75, METRICMM reports a return of 4.55 versus 0.06 for plain concatenation (Geometry of Uncertainty). This is the right competitor because it attacks the same corruption with upstream filtering instead of downstream decomposition. There is no reported result on Lacuna for DIPPER under any sensor corruption, and none for DIPPER combined with METRICMM, so we establish both ourselves.
A third baseline is DIPPER with heavy observation-noise randomization during training. It represents the standard sim-to-real practice.
A fourth is DIPPER with a scalar worker critic replaced by an ensemble, where ensemble disagreement scales the feasibility penalty. This is uncertainty handling without decomposition.
On hardware we add the threshold-switched hybrid controller. It is the strongest physical number in the file: 90% success and 10% collisions against 30% for the monolithic controller (Monolithic vs. hybrid).
Fairness rules apply to all baselines. Every method shares the same worker architecture size, the same preference dataset of roughly 10,000 comparisons per environment as in DIPPER, the same environment-step budget, the same hyperparameter-search budget of 20 trials each, and the same number of seeds.
Beating the METRICMM-fronted baseline requires two results. The first is a lower false-infeasibility rate. The second is higher task success on at least three of the four DIPPER environments under corruption, with collision rate no worse. Beating the hybrid on hardware requires matching its 90% success while reducing discontinuities, measured as jerk around subgoal switches.
Three ablations isolate the key ingredient:
- Decomposed but unrouted: two critics that both see all sensors. This tests whether routing, rather than decomposition itself, drives the effect.
- Routed with $c_t$ fixed at 1: this tests the confidence weighting.
- Multi-head with a shared trunk instead of disjoint critics: this tests the Multi-Critic Actor Learning finding in our setting.
Why not just…?
Why not just filter the observations with METRICMM and keep a scalar critic? This is the real threat. If filtering removes the corruption, decomposition adds nothing. We expect filtering to fail in two regimes. The first is persistent corruptions, such as a steady depth bias, that a dynamics model gradually accepts as consistent. The second is total dropout, where no clean estimate exists but the kinematic feasibility question is still answerable from proprioception. The combined condition, routed decomposition with METRICMM confidence, against METRICMM alone settles whether the gain is additive. If METRICMM alone matches us, we will report that as the result.
Why not just give the critic privileged clean simulator state during training? An asymmetric critic works when the value is only needed at training time. Here the feasibility penalty also filters subgoals at deployment, where no privileged state exists. We include a privileged-critic variant that must fall back on real observations at test time, to show where it breaks.
Why not just randomize sensor noise in training? METRICMM reports that noise injection fails when test corruptions differ from the training distribution. Our protocol applies held-out corruption families at test time, so the randomized baseline directly measures this.
Why not just use an uncertainty-aware ensemble of scalar critics? An ensemble can flag that the value is uncertain, but not why. It therefore down-weights reach and clearance together, and the manager loses its kinematic feasibility signal exactly when perception fails. The ensemble baseline tests whether this distinction matters in practice.
Why not just switch between two policies with a threshold? The Franka study shows this works on a single reach-and-avoid task. The same authors report chattering at the boundary, and DIPPER's results show that unconstrained high-level choices drive workers out of their operating envelope on long multi-stage tasks. Franka Kitchen is where we expect a hand-set threshold to have no natural analogue.
Decisive Checks
The first check is the diagnostic. If scalar-value DIPPER under corruption shows a false-infeasibility rate under about 5%, the proposal's premise fails.
The central check is a plot of false-infeasibility rate and task success against corruption severity, for each of the seven corruption families, across all baselines and ablations. The routed-decomposition curve should stay flat for kinematic feasibility while the scalar and unrouted curves climb.
The safety check is the collision rate on trials where corruption hides a real obstacle. If confidence-weighting raises collisions above the METRICMM baseline, the weighting rule must be replaced with a conservative one that shortens subgoals instead.
The last check tests whether $V_{\text{kin}}$'s missing obstacle information matters. We compare its feasibility judgments against oracle reachability in cluttered scenes. The proprioception-only critic should mislabel only subgoals that obstacles block, and those are the cases $V_{\text{ext}}$ is meant to catch.
References
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- Multi-Critic Actor Learning: Teaching RL Policies to Act with Style
- Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learning
- Hierarchical goals contextualize local reward decomposition explanations
- Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RL
- Monolithic vs. hybrid controller for multi-objective Sim-to-Real learning
- Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robot