When one policy network is trained on many reinforcement learning tasks at once, the tasks often pull its shared weights in opposite directions. The standard fixes, PCGrad and CAGrad, detect and remove these conflicts by comparing each task's gradient vector with the ordinary dot product. This proposal starts from a different comparison. A policy's natural gradient is its gradient rescaled by how much each parameter change actually alters the policy's action probabilities. Under that measure, the overlap between two tasks' update directions equals something simple and cheap to estimate: the average product of the two tasks' compatible critics, evaluated on the same state-action pairs.
A compatible critic is the linear value model from Sutton et al. that leaves the policy gradient exactly unbiased. Put plainly, two tasks conflict when their compatible critics disagree about which actions are good at the same states. We propose to measure conflict this way, then resolve it with a conflict-averse step computed in the policy's own geometry. The resulting multi-task natural actor-critic needs only a K-by-K matrix of critic-prediction products per update, where K is the number of tasks. It never inverts a parameter-sized matrix, and it never differentiates a deep critic with respect to actions.
Core Claim
Let $\psi_\theta(s,a)=\nabla_\theta \log \pi_\theta(a|s)$ be the policy's score, the direction in parameter space that raises the log-probability of action $a$ at state $s$. The first ingredient comes from Policy Gradient Methods for Reinforcement Learning with Function Approximation. That paper proved that a critic linear in $\psi$, $Q_w=w^\top\psi$, and fit by least squares, yields the exact policy gradient however poorly it fits the true values. The second ingredient comes from the natural actor-critic line summarized alongside Actor-Critic Algorithms: the fitted weights $w^*$ are the natural gradient $F^{-1}\nabla J$, where $F=\mathbb{E}[\psi\psi^\top]$ is the Fisher matrix.
We extend this to K tasks sharing $\theta$, where task $i$ is sampled under its own state distribution $d_i$ (and possibly its own dynamics). Pool all tasks' samples into the mixture $\bar d=\frac1K\sum_i d_i$ with Fisher $\bar F$. For each task $i$, regress on the pooled batch a target equal to task $i$'s advantage estimate on task-$i$ samples and zero on every other task's samples. The normal equations give $w_i=\bar F^{-1}\nabla J_i/K$, the natural gradient of task $i$ in the shared geometry. Consequently
$$ G_{ij}\;=\;w_i^\top \bar F\, w_j\;=\;\mathbb{E}_{(s,a)\sim \bar d}\!\left[(w_i^\top\psi)(w_j^\top\psi)\right]\;=\;\tfrac{1}{K^2}\,\nabla J_i^\top \bar F^{-1}\nabla J_j , $$
where $G$ is the K-by-K Gram matrix of task directions in the Fisher metric. Every entry is a batch average of products of two scalar critic predictions.
The off-diagonal entries have a concrete reading. $G_{ij}<0$ means that moving the shared policy toward task $i$'s preferred actions lowers the probability of actions that task $j$'s compatible critic favors, on the states where both tasks' data live. We then solve CAGrad's dual over task weights, which only needs inner products among task directions. The difference is that the constraint ball $\|d-\bar w\|_{\bar F}\le c\|\bar w\|_{\bar F}$ is measured in the Fisher metric, where $\bar w$ is the average of the $w_i$. The output is a combination $d=\sum_i\alpha_i w_i$ that is itself a natural-gradient step.
The claim to test is that conflict measured this way predicts real interference better than Euclidean cosine on deep-critic gradients. It should also inherit compatibility's insensitivity to critic misfit. In particular, $G$ is exact whenever the $w_i$ are least-squares optimal, independent of how well any deep value network fits.
Relation to Prior Work
Prior work plays three roles here. Some of it motivates the gap. Gradient Surgery for Multi-Task Learning projects away negative-dot-product components between task gradients. Conflict-Averse Gradient Descent for Multi-task learning maximizes the worst task's improvement inside a Euclidean ball around the average gradient, and shows that PCGrad and MGDA can settle at poor Pareto points. Both treat the RL objective as a black box and measure conflict with the Euclidean dot product on gradients that flow through deep critics. We keep CAGrad's dual machinery but change the metric, and we compute the inner products from compatible-critic predictions.
Other work constrains the design. Compatible Gradient Approximations for Actor-Critic Algorithms shows that the action derivatives of deep critics are noisy and biased. That warns against surgery on SAC-style gradients, and it is why we use score-function (likelihood-ratio) critics instead of action derivatives. CASA shows that critic fitting and actor improvement conflict even within one task, and aligns them by tying $\nabla_\theta Q$ to $\nabla_\theta\log\pi$. Compatibility gives that alignment by construction, which is why we build on it rather than on CASA's single-task heuristic.
A third group supplies the method. Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation proves that single-loop natural actor-critic with compatible critics removes the critic error floor and reaches a global optimum in $\mathcal{O}(\epsilon^{-3})$ samples. Its auxiliary-chain argument is the template for our analysis. Compatible natural gradient policy search shows that natural steps paired with compatible values need an entropy lower bound to avoid collapsing exploration, and we adopt that safeguard.
The closest multi-task work falls short in specific ways. Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning optimizes a shared policy for summed task returns with natural steps. However, it uses fixed linear features rather than score features, so it has a critic error floor. It averages tasks with no conflict handling, and it assumes identical dynamics, whereas our mixture Fisher construction does not. Online Multi-Task Learning for Policy Gradient Methods factors task policies through a shared basis but needs precomputed per-task Hessians and has no critic. Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic blends local and shared advantages with a fixed weight. That blend is not a least-squares projection onto the score, so it gives up compatibility, and it has no conflict signal.
Research Program
On the algorithm side, we will build Fisher-CAGrad, a multi-task natural actor-critic with a shared task-conditioned policy. The critic targets are generalized advantage estimates, one set per task. The K compatible regressions share one design matrix, so we solve them as a single multi-output least-squares problem. For small linear or tabular policies we form the problem explicitly. For neural policies we use matrix-free conjugate gradient with Fisher-vector products and K right-hand sides, as TRPO-style solvers do. The Gram matrix $G$ then comes from pooled predictions, the task weights come from CAGrad's K-dimensional dual, and the step is taken under a KL trust region with an entropy floor.
Cost is honest and roughly comparable to PCGrad. Solving for K right-hand sides scales linearly in K. For MT50 we will subsample tasks per update in the manner of CAGrad-Fast.
On the theory side, we aim for two results. The first is an exactness statement: $G$ equals the true Fisher Gram matrix of task gradients under the least-squares compatible fit, for any per-task value misfit, extending Sutton et al.'s orthogonality argument. The second is a convergence result. CAGrad proves that its update converges to a stationary point of the average loss for $c<1$ in the Euclidean case, and we will carry that argument into the Fisher metric. We will then combine it with the single-loop analysis of Wang et al. to bound sample complexity when the compatible weights are tracked by stochastic updates rather than solved exactly. We expect a rate no better than the single-task $\mathcal{O}(\epsilon^{-3})$, since per-task tracking error adds rather than cancels. The open risk is that the auxiliary-chain argument breaks when the mixture distribution shifts with $\theta$ across heterogeneous dynamics. If it does, we will state the result for shared dynamics first, which is FedNAC's setting.
Baselines
The naive reference point is multi-task SAC with a shared backbone and uniform loss averaging. Conflict-Averse Gradient Descent for Multi-task learning reports MT50 success of 0.36 for it. Meta-World reports 85% on MT10 and 40% on MT50 for multi-headed SAC. We rerun it under our protocol rather than copying these numbers.
The strongest existing method is CAGrad on multi-task SAC, with a reported MT50 success of 0.52 against 0.50 for PCGrad in the same paper. Gradient Surgery for Multi-Task Learning reports roughly 70% for PCGrad on MT50 under its own protocol. That discrepancy is itself a reason to rerun everything in one harness, possibly built on MTBench if it supports Meta-World. All methods will get the same environment-step budget, the same network width, the same seeds (at least ten), and an equal hyperparameter-search budget in trials.
Because our method is on-policy, comparing only against off-policy SAC would conflate the metric question with sample reuse. We therefore also run same-family baselines: multi-task natural actor-critic with averaged tasks (Fisher-CAGrad with $c=0$), and multi-task PPO with PCGrad and with CAGrad. No reported result was found on Lacuna for multi-task natural actor-critic or PPO with gradient surgery on Meta-World.
The ablation that isolates the key ingredient is Euclidean-CAGrad-on-natural-directions. It computes the same per-task natural gradients $w_i$ but compares them with the plain dot product $w_i^\top w_j$ instead of $G_{ij}$. It differs from our method only in the metric used to detect conflict. A second ablation swaps compatible score features for fixed features in the critic, as in FedNAC.
Beating the naive and same-family baselines by a clear margin on MT10 and MT50, and beating the Euclidean-metric ablation, is the core evidence. Matching or exceeding 0.52 on MT50 at an equal step budget would be a strong result. Losing to SAC-based CAGrad while beating on-policy CAGrad would still answer the metric question, but would say nothing about overall practicality, and we would report it that way.
Why not just…?
Why not just run PCGrad or CAGrad on multi-task SAC, which already works? It does work, reaching 0.52 on MT50, and it remains the bar for practical performance. The question here is narrower: is the Euclidean dot product on deep-critic gradients the right conflict signal? Saglam et al. show that the action derivatives feeding SAC's actor gradient are noisy and biased, so a measured cosine partly reflects critic error. Our controlled experiment below measures that directly by comparing estimated conflict with true conflict. If Euclidean surgery on SAC turns out to estimate conflict as accurately as $G$, the proposal's premise fails.
Why not just take natural gradient steps on the averaged objective? This is a real threat. Rescaling by the Fisher matrix already normalizes gradient magnitude, which is one leg of Yu et al.'s tragic triad, and FedNPG shows that averaged natural steps converge globally under shared dynamics. The $c=0$ ablation settles it. If averaged multi-task natural actor-critic matches Fisher-CAGrad on MT10 and MT50, then the conflict-aversion part adds nothing, and the contribution reduces to the cheap diagnostic $G$. We commit to reporting that outcome.
Why not just compute per-task natural gradients and apply ordinary CAGrad to them? This is the obvious combination, and it is our key ablation. The two methods agree only if $\bar F$ is close to a multiple of the identity. The Fisher metric counts two parameter changes as conflicting only if they move the policy's action probabilities in opposing ways on shared states. The Euclidean metric can flag conflicts in directions the policy barely responds to, and can miss conflicts concentrated in a few sensitive directions. We will report how often the two metrics disagree on the sign of $G_{ij}$ and whether those disagreements predict the per-task returns measured after each step.
Why not use Adam's diagonal preconditioning as a cheap stand-in for the Fisher metric? A diagonal approximation drops exactly the cross-parameter structure that couples tasks through shared layers. We include it as a variant (diagonal-Fisher Gram), so the benefit of the full metric is measured rather than assumed.
Proposed Experiments
The first decisive check runs where true gradients are computable. We use the decentralized multi-task gridworld of FedNAC, where tasks share dynamics but see different rewards, plus a variant with task-specific slip probabilities that breaks FedNAC's shared-dynamics assumption. We will also use Acrobot variants with different link masses, following the setting of Wang et al. In these environments the exact $\nabla J_i$ and $\bar F$ are available by dynamic programming or long Monte Carlo runs. We compare three conflict estimates against the true Fisher Gram matrix: our $G$, the Euclidean cosine of deep-critic actor gradients, and $G$ built from fixed-feature critics. We deliberately misspecify the value networks at increasing levels. The prediction is that $G$'s error depends only on regression sample size, while the other two degrade as critic misfit grows. If $G$ degrades just as fast, the exactness claim is wrong in practice.
The second check tests whether measured conflict predicts interference. After each update we record each task's one-step change in return. We then test whether negative $G_{ij}$ predicts return drops on task $j$ better than the Euclidean cosine does, scored by rank correlation across training.
The third check is the main benchmark. We run Meta-World MT10 and MT50 with the baselines above, using mean success rate over the final evaluation window as the headline metric. We also report worst-decile task success, since conflict aversion should mainly help lagging tasks, and wall-clock time per update to account for the cost of the K right-hand sides.
The fourth check probes heterogeneous state and action spaces. We use the 2-, 3- and 4-link Reacher and Insert suite from MTAC-SC, where the paper reports 95% success on 4-link Insert against a 60% single-task plateau. The comparison is whether a shared compatible policy with Fisher-CAGrad can match MTAC-SC's fixed advantage blending without tuning its blending weight.
The proposal is considered supported if $G$ tracks true conflict under critic misfit and predicts interference better than Euclidean measures. It also needs Fisher-CAGrad to beat both averaged natural actor-critic and the Euclidean-metric ablation on Meta-World. It fails if the $c=0$ ablation ties it.
References
- Policy Gradient Methods for Reinforcement Learning with Function Approximation
- Actor-Critic Algorithms
- Gradient Surgery for Multi-Task Learning
- Conflict-Averse Gradient Descent for Multi-task learning
- Compatible Gradient Approximations for Actor-Critic Algorithms
- CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation
- Compatible natural gradient policy search
- Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning
- Online Multi-Task Learning for Policy Gradient Methods
- Multi-task Actor-Critic with Knowledge Transfer via a Shared Critic
- Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning