Free Value-Gradient Labels for Multi-Task Critics from Goal Parameters

When one neural network learns the value of acting across many robot tasks at once, it sees each experience labeled with only a single number: an estimate of future reward. The policy, however, never uses that number directly. It uses how the value changes when the action or the target changes. This proposal gives a shared multi-task critic (the network that scores actions) a second, richer label at almost no cost: the derivative of its own training target with respect to the task's goal parameters, such as the target position of a reach or the destination of a push. In most manipulation suites the goal enters only the reward, not the physics. Because of that, this derivative can be computed exactly by automatic differentiation of the reward and the target network. No differentiable simulator is needed, and that requirement has so far confined first-order value learning to single-task toy and locomotion settings.

The bet is that these labels are especially valuable in the multi-task regime. A shared critic must keep many value landscapes apart, and the goal gradient directly describes how each task's landscape is shaped. It gives $d$ extra numbers per transition, where $d$ is the goal dimension. If the bet is right, compact critics should recover part of the benefit that large-scale work currently buys with billion-parameter networks. We also expect less of the cross-task gradient conflict that methods like gradient surgery try to repair after it has already happened.

Core Claim

In multi-task suites whose tasks carry continuous goal parameters that affect reward but not dynamics, first-order Bellman supervision in the goal variable is model-free and exact. Training a shared critic to match these goal gradients, with each task's gradient term normalized by its own gradient scale, should do three things relative to an equally sized scalar critic. It should improve multi-task success and sample efficiency. It should reduce measured conflict between per-task parameter gradients. Its advantage should shrink as critic capacity grows. The last prediction separates our account (the extra labels substitute for capacity) from a generic regularization story.

Concretely, for transition $(s, a, s')$ under task $k$ with goal vector $g$, reward $r_k(s,a,g)$, target critic $\bar Q$ and target policy $\pi$, the scalar target is $y = r_k(s,a,g) + \gamma \bar Q(s', \pi(s',g), g)$. Its goal gradient follows from the chain rule, because $s'$ does not depend on $g$:

$$\nabla_g y = \nabla_g r_k(s,a,g) + \gamma \, \frac{d}{dg}\bar Q(s', \pi(s',g), g).$$

The critic $Q_\phi$ is trained on

$$\mathcal{L}(\phi) = \mathbb{E}\big[(Q_\phi - y)^2\big] + \lambda \sum_k \frac{1}{\sigma_k^2}\,\mathbb{E}_k\big[\|\nabla_g Q_\phi - \nabla_g y\|^2\big],$$

where $\sigma_k^2$ is a running average of $\|\nabla_g y\|^2$ for task $k$ and $\lambda$ is a single weight shared by all tasks.

The normalization matters, and it is where we depart from the folk argument for value gradients. Gradients are invariant to constant offsets in value, as Improving Policies without Measuring Merits observes. They are not invariant to reward scale: a task whose rewards are ten times larger has goal gradients ten times larger. Gradient supervision alone therefore does not fix reward-scale dominance. The per-task normalization does that job, and we test it separately from the gradient labels themselves.

Relation to Prior Work

Two papers supply the method. Goal-Conditioned Q-learning as Knowledge Distillation (ReenGAGE) trains a goal-conditioned critic to match the goal gradient of its target network. It argues that this cuts sample complexity in goal dimension from $\mathcal{O}(d^2)$ to $\mathcal{O}(d)$ and shows faster learning than DDPG with hindsight relabeling on ContinuousSeek and HandReach. Its tasks differ only by goal coordinates inside one environment with one reward form. We carry the same signal into suites of physically distinct task families that share one network, where the open questions are interference and reward heterogeneity rather than goal dimension alone. First-order Sobolev Reinforcement Learning and CACTO-SL train critics on state and action derivatives obtained from differentiable simulators or trajectory-optimization backward passes. ("Sobolev" training means supervising a network on both its outputs and its derivatives.) CACTO-SL reports a 92% reduction in trajectory-optimization episodes on a double integrator. These two papers also constrain the design. Their derivatives require differentiable dynamics, which break down at contact, so we deliberately restrict ourselves to the goal variable, where contact never enters the chain rule. The Sobolev RL paper also reports that on MuJoCo Ant the gradient-trained critic learns faster early but reaches final reward comparable to standard soft actor-critic. A single-task gain may therefore not survive to the asymptote. We treat the multi-task setting as the place to test whether it does.

Three lines of work attack multi-task interference without value gradients, and they motivate the question. Gradient Surgery for Multi-Task Learning (PCGrad) projects conflicting per-task parameter gradients and lifts soft actor-critic on Meta-World MT50 from under 40% to about 70% success. RotoGrad rotates shared features per task to align gradient directions and magnitudes, but only on supervised vision benchmarks. Bigger, Regularized, Categorical (BRC) scales a normalized residual critic beyond a billion parameters and uses a categorical loss so that tasks with different reward scales produce comparable losses. All three act on the parameter update or on capacity. None changes what the critic is asked to fit. We change the supervision itself and then measure whether parameter-level conflict goes down as a consequence.

Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic shares a scalar critic across arm morphologies through zero-padding and a fixed advantage blend. It is the closest shared-critic design, but its critic receives scalar targets only.

Research Program

The first component is engineering. We will reimplement the Meta-World V2 reward functions in an automatic-differentiation library, starting with MT10 and extending to MT50. The suite will follow the standardized version in Meta-World+, because that paper shows reward-version differences alone move MT50 success from 61.4 to 83.2 for one method. We verify each reimplementation against the simulator's reward on logged transitions. For each task we also record where the gradient of reward with respect to the goal is zero or undefined: flat tolerance regions and branches that depend on whether the object is grasped. That record is our map of where the signal is informative.

The second component is the training method. We use a multi-task soft actor-critic with a shared trunk and a task one-hot, the setting PCGrad and the original benchmark use, and add the normalized goal-gradient loss above. The derivative of the target term includes the path through the target policy. We will ablate stopping the gradient at the policy and at the entropy term, since ReenGAGE-style targets are defined on the full derivative. Double backpropagation roughly doubles critic cost, so every comparison is matched on wall-clock time as well as on environment steps.

The third component is a measurement study. On fixed checkpoints we record two quantities. The first is the cosine similarity between per-task critic parameter gradients, the quantity PCGrad acts on. The second is the error of the critic's goal gradient against ground truth. We estimate the ground truth by finite differences of Monte Carlo returns under small goal perturbations, using identical simulator seeds, which is possible because MuJoCo is deterministic given the seed. This tells us whether better goal-gradient accuracy goes together with less parameter-gradient conflict, which is the mechanism the core claim asserts.

Baselines

The naive reference point is multi-task, multi-headed soft actor-critic with a scalar critic. Meta-World: A Benchmark and Evaluation reports 85% success on MT10 and 40% on MT50 for it. We rerun it in our codebase on the Meta-World+ V2 suite rather than relying on those numbers, because of the reward-version issue.

The strongest published Meta-World figure in the file is MOORE on MT50 under V2 rewards, at 83.2 success as reported by Meta-World+. SM reaches 84.9 on MT10 under V2 in the same paper. Our method adds a loss term rather than an architecture, so the fair test is to add the goal-gradient loss on top of whichever of these architectures we can reproduce. Beating the baseline means a gain beyond seed variance at matched steps and matched wall-clock.

PCGrad on a scalar critic is the strongest conflict-mitigation competitor. Its reported MT50 figure is about 70% success (Gradient Surgery), on the older reward version. We run it with the same network, the same replay, and an equal tuning budget of the same number of trials for its learning rate as we give to $\lambda$.

BRC is the capacity competitor. The file reports that it needs roughly 40 times fewer gradient updates per task than single-task specialists, but it gives no MT10 or MT50 success number. There is no reported result on Lacuna for BRC success on Meta-World MT10 or MT50. We therefore run a compute-feasible BRC-style critic (normalized residual blocks with categorical loss) at three widths, with and without our loss.

Two ablations isolate the key ingredient. The first keeps per-task normalization of the scalar TD loss but removes the gradient term, which separates rebalancing from new information. The second keeps the gradient term but feeds it targets computed for a randomly permuted goal from the same task family. This control has identical compute and identical smoothness pressure but wrong content. If it matches the full method, the gain is regularization, not information.

Why not just…?

Why not just normalize rewards per task or use a categorical loss? This is the most serious threat, because our own analysis says gradient labels do not fix reward scale by themselves. The rebalancing ablation tests it directly: per-task-normalized scalar loss, and separately a categorical critic in the style of BRC, against the same plus goal-gradient labels. If normalization alone closes the gap, the honest conclusion is that the earlier gains came from balancing, and the proposal shrinks to that negative finding. Because the categorical critic's expected value is differentiable in the goal, the two ideas combine, and the combination is a planned condition rather than a rival.

Why not just apply PCGrad to a scalar critic? PCGrad repairs conflicting updates after the loss has been computed and costs a backward pass per task. It does not give the critic any information about landscape shape. Our measurement study reports parameter-gradient conflict with and without goal-gradient labels. If conflict drops substantially under our loss, the two methods target the same failure from different ends, and PCGrad on top of our loss should then add little. If conflict does not drop, the methods are complementary, and we report the stacked result.

Why not just add hindsight goal relabeling or more critic updates? Relabeling also creates extra supervision along the goal dimension, and a higher update-to-data ratio uses the same extra compute that double backpropagation consumes. ReenGAGE reports that DDPG with hindsight relabeling fails to match goal-gradient matching on 20-dimensional ContinuousSeek even with twice the training steps (Goal-Conditioned Q-learning as Knowledge Distillation). That was a single environment, however. We include relabeling and a wall-clock-matched higher update ratio as baselines on Meta-World rather than assuming the result transfers.

Why not just make the critic bigger? BRC shows that capacity mitigates interference. Our claim is narrower: at the capacities practical for many labs and robots, goal-gradient labels buy a similar effect. The width sweep decides this. We predict the gap closes as width grows. If the gap is constant or grows, our "substitute for capacity" account is wrong even if the method helps.

Proposed Experiments

We first reproduce ReenGAGE on ContinuousSeek as an implementation check. The main study then runs on Meta-World+ V2 MT10, MT25 and MT50, which the benchmark paper reports at roughly 6, 12 and 25 hours of training respectively. We use at least five seeds per condition and report average success, area under the success curve over environment steps, and success at matched wall-clock time.

Three results are decisive. First, the full method must beat both the normalized-scalar ablation and the permuted-goal control on MT10 and MT50. Otherwise neither information nor balancing from the gradient labels is doing the work. Second, the per-task gain should track the fraction of transitions in that task with nonzero, well-defined reward-to-goal gradients, taken from our reward map. Tasks dominated by flat or branching rewards should gain least. This is a falsifiable prediction tied to the mechanism. Third, across critic widths, the improvement should shrink as width grows, alongside a measured drop in per-task parameter-gradient conflict and in finite-difference goal-gradient error.

A positive outcome would establish goal-gradient supervision as a cheap, simulator-free addition to multi-task critics. A negative outcome on the first check would still settle an open question: the scale-invariance argument often made for value-gradient learning does not, on its own, address multi-task reward imbalance.