Factored Critics for Multiplicative Rewards in Hierarchical Humanoid Control

When a simulated humanoid learns to lift a barbell or pick up a basketball from human motion capture, the reward usually has several parts: match the human's posture, move the object along the demonstrated path, and make the right body-object contacts. Recent work found that adding these parts together lets the agent "hack" the reward. It keeps its posture perfect and ignores the ball. Multiplying the parts instead forces all of them to be satisfied at once. The cost is that early in training at least one part is almost always near zero, so the product is near zero, and the agent gets no learning signal. Existing systems escape this stall with hand-built fixes, such as injecting expert demonstrations into training.

We propose to keep multiplication as the definition of success, while never asking a network to learn from the product directly. The low-level controller learns a separate value estimate for each reward factor, and each factor has its own dense signal. A high-level controller picks time-extended subgoals and weights for the low level. It scores each candidate by the predicted product of the per-factor values, and it shifts attention toward whichever factor is currently weakest. If this works, the two-level structure becomes the mechanism that turns a vanishing all-or-nothing reward into learnable pieces, rather than an architectural add-on.

Core Claim

The vanishing-signal failure of multiplicative rewards is a credit-assignment problem, and a temporal hierarchy with factored values solves it more reliably than demonstration injection or a flat log-transformed reward. Concretely, suppose the task reward is $R_t = \prod_{i=1}^{k} r_{i,t}$ with each factor $r_i \in [0,1]$. Train the low-level worker with $k$ critics $V_i(s, g)$, one per factor, each regressing only its own factor's discounted return under subgoal $g$. The manager then selects subgoals by maximizing

$$ J(g) = \sum_{i=1}^{k} \lambda_i \log V_i(s, g), $$

where the weights $\lambda$ lie on the probability simplex. They are updated by mirror descent toward the factor with the lowest current value. This update is the max-min reweighting of Fast RL, moved from flat language-model training to the subgoal level of a robot hierarchy. With uniform $\lambda$, $J$ is the log of the predicted product, so the manager still optimizes the all-or-nothing objective. The low-level policy receives the same $\lambda$ as an input and is trained on the $\lambda$-weighted sum of per-factor advantages. Every factor critic therefore keeps learning even while the product is zero. Evaluation always uses the true product and task success, so any hacking of individual factors is visible.

Where This Sits

Three papers motivate the problem. SkillMimic multiplies posture, ball-trajectory and contact-graph rewards. It reports 86.7% basketball pickup success against 19.6% for DeepMimic and 0% for AMP, and it already uses a high-level controller over learned skills. However, all the multiplicative factors live inside the low-level skill, and the high level sees only a scalar. PMP multiplies style rewards from separate adversarial motion priors for each body part. AMP is a discriminator that rewards motion resembling human clips. PMP reports a normalized return of 0.55 on barbell lifting versus 0.022 for AMP. It also states that it needed "Demo Blend", stochastic injection of reference trajectories, to avoid collapse when the product goes to zero. That heuristic is the direct target of this proposal. Redundancy Resolution as Action Bias documents the opposite failure of additive weighting: high secondary weights make the agent timid, and low ones make it ignore secondary goals.

Two papers supply the method. DIPPER regularizes a high-level policy with the low level's scalar value, so that it does not propose unreachable subgoals. Its ablation without this term drops significantly. We reuse the idea that the manager should consult the worker's values, but we make the value vector-valued and composed multiplicatively. We do not use preference optimization. Fast RL supplies the mirror-descent max-min weighting. It was shown only on flat text generation, for example a 66% win rate versus 10% for PPO on DailyDialog, so whether it behaves well in a hierarchy with physics is an open question we test.

One paper constrains the design. Puppeteer shows that a low level pre-trained on motion capture gives natural movement without hand-tuned style rewards. This raises the question of whether a strong frozen motion prior alone removes the stall. We therefore build on a motion-capture-pretrained low level and test this directly, rather than assuming it does not.

Research Program

We build a two-level controller for a simulated humanoid. The low level is initialized from motion-capture tracking, following the Puppeteer recipe, and runs at control frequency. It is conditioned on a subgoal (target keypoints for hands, head and feet plus a target object state) and on the weight vector $\lambda$. It has $k$ critic heads, one per factor. In the first set of tasks the factors are posture, object trajectory and contact pattern, taken from SkillMimic. In a later handover variant we add a clearance-to-human factor. The manager acts every few hundred milliseconds. It samples candidate subgoals from a learned proposal distribution, scores them with $J(g)$, and updates $\lambda$ by mirror descent on per-factor running returns.

The analysis has three parts. The first is diagnostic. We measure, during training, the fraction of steps where the true product is below a small threshold, and the gradient magnitude reaching the low-level policy. We expect the product to be near zero for long stretches early on while factored advantages stay informative. The second is a hacking audit. We track each factor separately at evaluation time to check that max-min weighting does not let one factor saturate while another stays near zero. The third is a stress test of the weighting rule. We vary the number of factors from two to five, because Fast RL and Dual-Objective RL with HJB formulations report that Lagrangian-style weight updates oscillate beyond two objectives. Mirror descent may or may not avoid this in control.

Baselines

The naive reference is a flat policy gradient agent (PPO) with the raw multiplicative reward, the same motion-capture-pretrained initialization, and no demonstration injection. It shows how severe the stall is in our setup. Alongside it we run the same flat agent with a tuned additive weighted sum. This shows the hacking side of the trade-off, measured by per-factor success.

The strongest existing methods are PMP with Demo Blend and SkillMimic's multiplicative skill training with its high-level controller. These are the published answers to exactly this problem. Their benchmarks are custom, so we reimplement both inside our simulator and task suite. All methods get the same demonstration clips, simulation step budget and hyperparameter search budget. Demo Blend gets its demonstrations, and our method gets the same clips only through the motion prior. As sanity anchors we report the published figures: PMP's 0.55 normalized return on barbell lifting and 0.63 on rope climbing (PMP), and SkillMimic's 86.7% pickup success (SkillMimic). Beating them means matching or exceeding their final true-product return and success with fewer environment steps, without injecting demonstrations, and with lower variance across seeds.

We also run a staged-reward baseline in the style of DexH2R (Zhao et al.), which switches from a reaching reward to a trajectory reward. It represents the hand-designed way of ordering factors in time.

Three ablations isolate the key ingredients. The first is a flat agent with the same factored critics and max-min weighting but no manager. It tests whether the hierarchy matters at all. The second is a hierarchy whose manager sees only a single critic trained on the product, which is DIPPER-style scalar value consultation. It tests whether factoring matters. The third is our full method with $\lambda$ frozen uniform. It tests whether max-min reweighting matters.

Why not just…?

Why not just take the logarithm of the product in a flat agent? The sum of $\log(\epsilon + r_i)$ never vanishes, and a near-zero factor gives a large penalty. This is the most serious threat. It may fix the stall on short tasks by itself. Its weaknesses are a choice of $\epsilon$ that trades signal against hacking resistance, and one critic that must fit a sum of terms with very different scales. Our flat factored-critic ablation sits between this baseline and our method. If log-sum matches our full method on every task, the honest conclusion is that the hierarchy is unnecessary and the contribution reduces to a reward transform. We will report that outcome as such. We expect the gap to appear on longer, sequential tasks such as carrying an object up stairs, where factors are satisfied at different times.

Why not just keep Demo Blend? It works on PMP's tasks, but it needs demonstrations covering every phase of the task. It also contaminates on-policy data with off-policy reference states, and PMP offers no account of when the injection rate can be lowered. Running it under the same budget in our suite measures whether a demonstration-free mechanism can match it. If it cannot, we have learned that demonstrations carry information that factored values do not recover.

Why not just rely on a strong motion-capture prior, as Puppeteer does? A good tracking low level may already produce non-zero posture and contact factors from the start, which would make the stall disappear. Puppeteer is evaluated on locomotion without object contact, so the file contains no evidence either way for manipulation. Our naive baseline uses the same prior, so the diagnostic measurements answer this directly. If the near-zero product fraction is already small with the prior alone, the motivation weakens and we will say so.

Proposed Experiments and Decisive Checks

The main suite reimplements barbell lifting, rope climbing and upstair carrying in the style of PMP, plus ball pickup and a dribble-then-shoot sequence in the style of SkillMimic. We use the BallPlay-M motion capture and GRAB for object-interaction references. GRAB lets us report ball position error against the published 78.0 mm from SkillMimic. The metrics are task success, normalized true-product return, per-factor success rates, environment steps to reach 80% of final performance, and seed variance.

A secondary transfer study adds a clearance factor and uses object and hand trajectories from the DexH2R handover dataset (DexH2R benchmark), replayed in simulation as moving targets. The published reference points are not reinforcement learning results in this setting, so we cite them only for context. The autoregressive MotionNet reaches 71.1% success but only a 25.9% safety rate in Easy Mode, and DP3 reaches a 33.7% safety rate in Hard Mode. No reported RL result was found on Lacuna for simulated replay of DexH2R trajectories. Our numbers there will be new, and they are not head-to-head comparisons.

The decisive check is a three-way comparison on the longest sequential task: the full method, the flat factored-critic ablation, and flat log-sum. The hierarchy claim stands only if the full method clearly beats both on true-product return under equal compute. The factoring claim stands only if the full method also beats the scalar-critic hierarchy. If both hold and Demo Blend is matched without demonstrations, the proposal has shown that a temporal hierarchy is a principled replacement for the heuristics that multiplicative rewards currently need.