Some goals for a robot hand are not sums of per-step rewards. "Learn to put this object into every orientation you can reach" and "make the object end up in poses distributed like this target distribution" are statements about the whole distribution of states the agent visits. The theory for such goals exists. It optimizes a non-linear function of the agent's occupancy measure, the long-run frequency with which it visits each state. That theory has only been run on small gridworlds, because it requires estimating that visitation distribution, and for a 48-joint pair of hands the full state has hundreds of dimensions. Hierarchical control offers a way around this. A high-level "manager" policy picks targets every few dozen steps, and a low-level "worker" policy moves the fingers to reach them. The manager's decisions live in a small space, such as the object's 7-number pose, where visitation distributions can actually be estimated.
The proposal is to apply the non-linear objective only to the manager's low-dimensional visitation distribution. The worker stays an ordinary goal-reaching learner with an additive reward. This placement has two consequences, and we will test both. First, the manager's non-linear objective becomes estimable in dexterous settings where a flat version is not. Second, because the worker keeps a standard Bellman-consistent value function, that value function can still act as a feasibility check on the manager's proposed targets. We test this on the Bi-DexHands bimanual benchmark, where existing methods learn isolated skills but fail when many tasks must be handled together.
Core Claim
Reinforcement learning with general utilities (RLGU) replaces the usual objective $\langle \rho_\pi, r\rangle$ with $f(\rho_\pi)$, where $\rho_\pi$ is the state-action occupancy measure of policy $\pi$ and $f$ is non-linear, for example concave. Three papers supply the method we reuse. Policy Gradient for Reinforcement Learning with General Utilities derives a policy gradient theorem that uses the "shadow reward" $\nabla f(\rho_\pi)$, a surrogate reward given by the local slope of $f$ at the current occupancy. Variational Policy Gradient Method for Reinforcement Learning with General Utilities turns concave $f$ into a primal-dual problem, proves global convergence, and evaluates only on FrozenLake. Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space reduces variance in the estimates and treats larger state-action spaces. All three assume a flat policy, and all three need an occupancy estimate over the space where $f$ is defined.
We define the utility over the manager's macro-occupancy $\rho_H$, the distribution of object configurations $\sigma$ at the manager's decision points, which occur once every $\tau$ primitive steps. The manager then maximizes the shadow reward plus a feasibility term:
$$\max_{\pi_H}\; \mathbb{E}_{\pi_H}\Big[\sum_k z(\sigma_k, g_k) + \beta\, V^{\pi_L}(s_k, g_k)\Big], \qquad z = \nabla f(\hat\rho_H),$$
Here $g_k$ is the target pose chosen at decision $k$, $\hat\rho_H$ is a slowly updated density estimate over the low-dimensional $\sigma$, $V^{\pi_L}(s,g)$ is the worker's value for reaching $g$ from state $s$, and $\beta$ weights the feasibility term.
The worker $\pi_L$ maximizes an ordinary additive goal-reaching reward. We train it with Heterogeneous-Agent Soft Actor-Critic, where each finger group is a cooperating agent. This follows Maximum Entropy Heterogeneous-Agent Reinforcement Learning, which learns useful Bi-DexHands policies in 10 million steps, a budget in which MAPPO learns nothing meaningful. Because the worker's reward is linear, $V^{\pi_L}$ satisfies the Bellman equation. It is therefore a legitimate feasibility critic of the kind Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach relies on.
This addresses a problem the research file raises: under a non-linear objective, the lower level's value function stops obeying the Bellman equation, so it cannot serve as a feasibility check. Our design avoids the problem by construction rather than by solving it.
We study two utilities. The first is coverage: $f = \mathcal{H}(\rho_H)$, the entropy of achieved object poses, whose shadow reward is $-\log \hat\rho_H - 1$. The second is target matching: $f = -\mathrm{KL}(\rho_H \,\|\, \rho^\star)$ for a user-specified non-uniform pose distribution $\rho^\star$. The second is where a pure novelty bonus cannot express the objective.
Relation to Prior Work
Several papers sit near two of the three directions but not all three.
MetaCURL: Non-stationary Concave Utility Reinforcement Learning has a "hierarchy," but it is an ensemble of algorithm instances used to minimize regret. It assumes known dynamics and runs many parallel instances. That is infeasible when a single dexterous run already costs billions of steps: ADEPT reports about 3 billion post-training steps. We borrow nothing from MetaCURL's machinery. It constrains our design, because it tells us not to track non-stationarity with ensembles.
DIPPER supplies the feasibility regularizer. We differ from it in three ways. It ranks subgoals with an offline preference set of about 10,000 pairs labeled by Euclidean goal distance. It is tested on 7- and 9-joint arms. Its objective is not distributional.
HASAC and Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning supply the platform and the motivating failure. Bi-DexHands collapses in multi-task and meta-learning settings: ProMP scores "a normalized score near zero" on ML20. HASAC adds per-step action entropy, which keeps the Bellman recursion and says nothing about which states the agent visits.
Multi-Agent Reinforcement Learning with General Utilities via Decentralized Shadow Reward Actor-Critic spreads shadow rewards across agents. We instead spread the objective across time scales and leave the finger-level agents with linear rewards.
Research Program
We will build a two-level controller for the dual Shadow Hands in Bi-DexHands.
The manager acts every $\tau$ steps. We sweep $\tau \in \{10, 25, 50\}$. At each decision it outputs a target object pose and, for bimanual tasks, a target relative pose between the hands.
The worker is goal-conditioned HASAC, trained with hindsight relabeling on the object-pose goal.
The density estimate $\hat\rho_H$ uses a discretization of position crossed with SO(3), the space of 3D rotations. A kernel estimate serves as a robustness check. The estimate is refreshed on a slower timescale than the manager's policy updates, which is the two-timescale scheme the RLGU convergence results require. The primal-dual step-size ratio is a known fragility of saddle-point methods, so we will report sensitivity to it explicitly rather than tune it quietly.
Training proceeds in two phases, both reusing the same trained worker. In phase one, the manager optimizes a coverage or target-matching utility with no task reward, producing a worker that can reach a wide set of object configurations. In phase two, the MT20 tasks are learned by a task-conditioned manager whose utility combines task success with coverage.
The analysis has three parts. We measure how accurately the shadow-reward gradient is estimated as a function of the occupancy space's dimension. We check whether the worker's value predicts actual reach success under the manager's shifting target distribution. We measure how often the manager proposes infeasible targets with and without the $\beta$ term.
Baselines
The naive reference point is flat PPO with the benchmark's handcrafted shaped reward, the recipe that masters single Bi-DexHands tasks. It shows what hand-designed linear rewards buy.
The strongest existing flat method is HASAC, as in Maximum Entropy Heterogeneous-Agent Reinforcement Learning. It is the strongest competitor on this platform because it is the method with demonstrated sample efficiency on the same 48-joint hands. The file gives no per-task success number for HASAC on Bi-DexHands, only the qualitative report that it learns useful policies within 10 million steps while MAPPO fails. For MT20 and ML20, the only figure is ProMP's near-zero normalized score on ML20. We found no reported MT20 or ML20 result on Lacuna for HASAC or for any hierarchical method. We will therefore rerun HASAC ourselves on MT20.
Beating HASAC means a higher MT20 normalized score at equal environment steps, averaged over five seeds, with non-overlapping 95% intervals. For the coverage experiment, it means more distinct orientation bins reached at equal steps.
Several baselines isolate the method's ingredients. A flat shadow-reward HASAC applies the same utility to the full state occupancy, estimated with the same density family. This isolates placement. A hierarchical controller whose manager receives a count-based novelty bonus over the same object-pose space isolates the utility form. A hierarchical controller with $\beta = 0$ isolates the feasibility critic. A goal-conditioned worker trained on uniformly sampled target poses, with no learned manager, is the reposing-style recipe that ADEPT's pre-training resembles.
All baselines share the same simulator, the same 2,048 parallel environments, and the same step budget of 200 million steps per run. At the reported throughput of more than 30,000 frames per second on one RTX 3090, a run takes a few hours. Every method, including HASAC, gets the same hyperparameter search of 20 configurations.
Why not just…?
Why not just hand-shape a linear reward with a coverage or diversity term? A per-step bonus cannot represent an entropy or KL objective over the visitation distribution, because the gradient of such an objective depends on the whole distribution and changes as the policy changes. Variational Policy Gradient makes this non-decomposability explicit. The shaped-PPO baseline measures the practical cost. The target-matching utility is a case where no fixed per-step reward is correct.
Why not just add a novelty bonus to flat HASAC? This is the most serious competitor, because the entropy shadow reward is itself a novelty bonus. Our claim is that where the bonus is computed matters more than its exact form. A density over hundreds of joint and contact dimensions is nearly uninformative. A density over object pose is not. Two ablations settle the question. The flat-versus-manager placement comparison tests where the bonus lives. The manager-level count-bonus ablation tests its form. If the manager-level count bonus matches the shadow reward on coverage, we will say so plainly and rest the utility-specific claim on the target-matching experiment, which a count bonus cannot express.
Why not just sample target poses uniformly, as in ADEPT-style reposing pre-training? ADEPT shows that goal-conditioned reposing across shapes transfers well, so uniform goals are a real threat for coverage. Uniform sampling spends effort on goals the hands cannot reach, and it cannot target a non-uniform distribution. The uniform-goal baseline measures both effects directly: fraction of infeasible goals, and KL to $\rho^\star$.
Why not just imitate demonstrations with a diffusion policy? Flat imitation needs demonstrations of the target behavior. For "every reachable orientation," such demonstrations do not exist. On the real bimanual DECO-50 data, DECO reports Diffusion Policy at 51.25% and ACT at 57.25% average success, and notes that flat models struggle on long tasks. That is a different setting from ours, so we cite it only as evidence that flat imitation is not a solved route for long-horizon dexterity. We do not run it here.
Proposed Experiments
The first experiment is coverage on single-object in-hand reorientation tasks from Bi-DexHands. We report entropy of achieved poses, the number of distinct SO(3) bins reached, and steps needed to reach 80% of the bins covered by the best method. The decisive prediction is a gap that grows with dimension. As we move from one hand to two hands to bimanual relative pose, flat shadow-reward HASAC should degrade while the manager-level version should not. The gradient-accuracy measurements should explain the gap.
The second experiment is target matching. We specify $\rho^\star$ as a mixture concentrated on hard-to-reach orientations and report $\mathrm{KL}(\rho_H \| \rho^\star)$. Here the count-bonus and uniform-goal baselines should fail by construction. If they do not, the utility-theoretic part of the claim is wrong.
The third experiment is MT20. We report normalized score at 200 million steps, comparing coverage-pretrained hierarchical agents against HASAC and PPO trained from scratch.
The $\beta = 0$ ablation provides the fourth check. We report the infeasible-target rate and success rate. The prediction is a drop matching DIPPER's finding that removing value regularization hurts, now under a non-linear manager objective.
If manager-level placement does not beat flat placement on coverage, the central claim fails regardless of the MT20 results.
References
- Policy Gradient for Reinforcement Learning with General Utilities
- Variational Policy Gradient Method for Reinforcement Learning with General Utilities
- Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
- Maximum Entropy Heterogeneous-Agent Reinforcement Learning
- Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
- MetaCURL: Non-stationary Concave Utility Reinforcement Learning
- ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning
- Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning
- Multi-Agent Reinforcement Learning with General Utilities via Decentralized Shadow Reward Actor-Critic
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter