Multi-task learning trains one shared network on several tasks at once. A popular family of optimizers treats disagreement between tasks as harm. When two tasks' gradients point in opposing directions, these optimizers project or constrain the update so the tasks stop fighting. A separate body of theory makes a seemingly opposite claim. A shared representation can only be learned in full if the training tasks are diverse, meaning they pull in many different directions so that every axis of the underlying feature space gets exercised. This note asks whether the conflict-removing optimizers, by suppressing disagreement, quietly remove the very diversity that lets a representation transfer to new tasks.
The core idea fits in two sentences. Opposing task gradients are often a symptom of task diversity rather than a pathology, so methods that delete them may score well on the training tasks while learning a narrower representation that transfers worse to held-out tasks. We propose to measure this effect directly, starting in a controlled linear setting where the true shared subspace is known and moving to deep benchmarks with held-out tasks. We then test a minimal fix: resolve conflicts only along feature directions the model has already learned, and leave untouched the components that point into directions it has not yet covered.
Core Claim
The theory of task diversity is precise about what it requires. Provable Meta-Learning of Linear Representations studies linear regression tasks that share a hidden $r$-dimensional subspace inside a $d$-dimensional input space. It shows that the subspace is recovered only when the task parameter vectors spread across all $r$ directions, so that their covariance has a minimum eigenvalue bounded away from zero. Few-Shot Learning via Learning the Representation, Provably states the analogous "covariance dominance" condition: the source tasks must cover every feature direction a target task needs. MAML and ANIL Provably Learn Representations shows that gradient-based meta-learning succeeds because per-task gradients, aggregated across tasks, jointly span the subspace. All three papers motivate this proposal, because they locate the benefit of multiple tasks in the spread of their signals.
Now consider a task distribution that is diverse in exactly this sense, with task vectors scattered roughly symmetrically around the origin of the shared subspace. Many task pairs then have negatively aligned parameters. Their gradients on the shared layers will often have negative inner products, which is precisely the condition that Gradient Surgery for Multi-Task Learning (PCGrad) detects and projects away. Conflict-Averse Gradient Descent for Multi-task learning (CAGrad) instead restricts updates to a ball around the average gradient while maximizing the worst-case improvement across tasks. Both methods are evaluated almost entirely on the training tasks themselves, using the average per-task drop relative to single-task models on NYUv2 or the success rate on Meta-World MT50.
Our claim is that this evaluation is blind to a real cost. Conflict removal can improve training-task metrics while lowering the coverage of the learned representation, defined as how much of a held-out task's useful signal lies inside the subspace the backbone actually learned. Prior work points the same way without testing it. ForkMerge argues that gradient conflict is not the same as negative transfer, noting that L2 regularization creates persistent conflicts while helping generalization. ForkMerge still measures only target-task accuracy, and it says nothing about the dimensional span of the representation. To our knowledge, nobody has asked what gradient surgery does to the subspace that downstream tasks inherit.
Research Program
The first stage is a controlled study in the linear multi-task regression model of Provable Meta-Learning of Linear Representations. We use their simulation scale: $d=100$, $r=5$, and $n=25$ samples per task. The model is a shared matrix $B \in \mathbb{R}^{d\times r}$ with a separate head per task. Here every quantity is observable. We can compute the principal angles between the learned column space of $B$ and the true subspace, track which true directions are learned and in what order, and evaluate excess risk on fresh target tasks. We will vary how the task vectors are spread inside the subspace: isotropic, concentrated in a few directions, and symmetric versus one-sided. We then compare plain averaged gradient descent, PCGrad, and CAGrad.
Two outcomes are possible. The hypothesis predicts that, under the projection, directions supported mainly by opposing task pairs are learned more slowly or not at all. It may instead turn out that the exact-subspace model is benign and the projection only reorders learning. Either result is informative. We will also attempt a short analysis of the expected PCGrad update in this model, since its gradient with respect to $B$ has a simple closed form in the task heads. The aim is to characterize when the projection removes subspace-spanning components and when it does not.
The second stage measures coverage in deep networks. For a held-out task, we fit a linear probe on the frozen shared features using all available labels. We then refit that probe using only the top-$k$ eigendirections of the feature covariance on the training tasks, and record the fraction of probe performance retained as $k$ grows. A curve that saturates quickly means the held-out task lives in the dominant learned directions. A curve that needs many directions, or never recovers, means it relies on directions the training barely shaped. We report this coverage curve alongside the effective rank of the features. Effective rank alone cannot separate useful spread from noise, and the experiments in stage four use exactly that distinction.
The third stage tests a minimal intervention, which we call coverage-gated surgery. It operates on gradients with respect to the shared features $z \in \mathbb{R}^p$, the same level at which RotoGrad applies its task-specific rotations. Let $C$ be a running estimate of the covariance of $z$ on the training tasks. Let $P$ project onto its top-$k$ eigenvectors. Let $\mathrm{Surg}(\cdot)$ be the PCGrad projection, applied across tasks. The update for task $i$ is
$$\tilde g_i = \mathrm{Surg}(P g_i) + (I-P)\, g_i,$$
where $g_i$ is task $i$'s gradient with respect to $z$. In words, conflicts are resolved only inside directions the representation already uses. Components pointing into new directions pass through unchanged, even when two tasks disagree there. Gating at $k=p$ recovers PCGrad, and gating at $k=0$ recovers plain averaged gradient descent. A single sweep over $k$ therefore interpolates between the two hypotheses. The method costs one eigendecomposition of a $p\times p$ matrix every few hundred steps, on top of PCGrad's per-task gradients.
Baselines
The naive reference point has two parts. The first is single-task learning on the held-out task from scratch. The second is a linear probe on a randomly initialized backbone. Together they tell us whether any multi-task representation helps a new task at all. That question matters because Provable Meta-Learning of Linear Representations proves a threshold below which multi-task features hurt.
Vanilla multi-task training with equally weighted losses is the most important comparison for our hypothesis. It preserves every conflicting component, so the hypothesis predicts it should transfer at least as well as surgery even where it scores worse on the training tasks. On NYUv2, CAGrad reports that the vanilla MTAN model has an average per-task drop of 5.59% relative to single-task models.
PCGrad and CAGrad are the methods whose side effect we are testing, and CAGrad is the strongest gradient-manipulation competitor. CAGrad reports an average per-task drop of 0.20% on NYUv2, against 3.97% for PCGrad, and an MT50 success rate of 0.52, against 0.50 for PCGrad and 0.36 for multi-task SAC.
ForkMerge is the strongest training-task competitor in the file. It reports a 4.03% relative improvement over single-task learning on NYUv2 and a 2.00% gain on DomainNet. Because it argues that conflicts are not the problem, it is the natural rival explanation for any effect we find.
For all held-out-task coverage and probe metrics, we found no reported result on Lacuna for any of these methods. We will therefore re-run every baseline ourselves. Each method gets the same backbone, data, and number of training steps. Each also gets the same hyperparameter budget, roughly twenty trials per method, chosen on a validation split of the training tasks only, never on held-out tasks.
Two ablations isolate the gating ingredient. In the first, $P$ is replaced by a random $k$-dimensional projector of the same rank. In the second, $P$ is inverted, so surgery acts on the uncovered directions. We will count coverage-gated surgery as beating the field only if it meets two conditions. It must stay within one standard deviation of the best training-task score among PCGrad, CAGrad, and ForkMerge. It must also improve held-out-task probe performance in the few-label regime over all three, across at least three seeds.
Why not just…?
Why not just use vanilla multi-task training, since it keeps all the conflicts? This is a real threat. If our hypothesis is right, vanilla training may already be the best choice for transfer, and the contribution would then be the diagnosis rather than a new optimizer. The deciding evidence is the comparison between the two. Vanilla training is clearly worse on training tasks on NYUv2 (5.59% drop) and MT50 (0.36 success), so the useful question is whether some setting of $k$ keeps most of surgery's training-task gain without the transfer loss. If no $k$ dominates both endpoints, we will report that surgery and vanilla training sit on a tradeoff curve and stop there.
Why not just add a rank or decorrelation regularizer on the features? Spreading out the feature spectrum is cheap, and it might seem to restore coverage. The theory, however, asks for coverage of task-relevant directions, not high rank in general. Excess risk bounds for multitask learning with trace norm regularization finds that even a capacity-shaping regularizer helps only when the inputs are spread across the feature space. The linear stage settles this, because there the true subspace is known. A rank regularizer that raises effective rank without shrinking principal angles to the true subspace fails our hypothesis's test. We include PCGrad combined with a rank regularizer as a baseline. If it matches gated surgery on held-out coverage in the deep experiments, we will recommend it as the simpler fix.
Why not just use ForkMerge, which already treats conflicts as harmless? ForkMerge picks merge weights on target validation data. It optimizes for the tasks being trained, not for tasks not yet seen, and averaging parameters could prune auxiliary directions just as projection might. We will measure its coverage curves directly. If ForkMerge preserves coverage while improving training tasks, then the parameter-space view explains our observations better than the gradient-diversity view, and we will say so.
Proposed Experiments
The decisive synthetic check uses the linear testbed described above, with $d=100$, $r=5$, $n=25$, and a sweep over the number of training tasks. We will plot the principal angle to the true subspace and the excess risk on new tasks as functions of the task count, for vanilla training, PCGrad, CAGrad, and gated surgery. The hypothesis predicts a gap in the symmetric-spread condition and none in the one-sided condition, where conflicts are rare. If no gap appears in either condition, the central claim fails in the setting where theory is clearest, and we will not pursue the deep experiments further.
The deep experiments all use held-out tasks. On the Taskonomy dataset, we train jointly on a subset of the 26 tasks and probe the rest. We then repeat with leave-several-out splits chosen two ways: by the Taskonomy affinity structure, so the held-out tasks are dissimilar to training, and at random. On CelebA's 40 attributes, used by RotoGrad, we train on 30 attributes and probe 10, with probe label budgets running from very few examples to the full set. On Meta-World, we train on a subset of the MT50 tasks and measure few-episode adaptation success on the remaining ones. Throughout, we report the standard NYUv2 average per-task drop as a sanity check that our re-implementations reproduce published training-task behavior.
Three results would be decisive. The first is whether conflict-removing methods show lower held-out coverage and few-label probe accuracy than vanilla training, despite better training-task scores. The second is whether the size of that gap tracks how often pairs of training-task gradients conflict, which the diversity account predicts and the distribution-shift account of ForkMerge does not. The third is whether the $k$ sweep of gated surgery traces a curve that strictly beats the line between its two endpoints, PCGrad and vanilla training. A positive answer on all three would give multi-task optimization a missing evaluation axis and a cheap way to protect transfer.
References
- Provable Meta-Learning of Linear Representations
- Few-Shot Learning via Learning the Representation, Provably
- MAML and ANIL Provably Learn Representations
- Gradient Surgery for Multi-Task Learning
- Conflict-Averse Gradient Descent for Multi-task learning
- ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Excess risk bounds for multitask learning with trace norm regularization
- Taskonomy: Disentangling Task Transfer Learning