Partial Task Affinity: Grouping Tasks by Direct, Not Shared, Gradient Agreement

When one neural network is trained on several tasks at once (multi-task learning), some tasks help each other and some interfere. Practitioners decide which tasks to train together by measuring how well task gradients agree, meaning whether a step that lowers one task's loss also lowers another's. These measures are pairwise. When every task depends on the same generic features, such as edges in images or daily cycles in time series, any two tasks look compatible simply because both want those features learned. That is the same trap as reading a correlation between two variables that are both driven by a third. Statisticians handle it with partial correlation, which measures the association that remains after regressing out the conditioning variables.

This note proposes computing task affinity as a partial correlation among task gradients. Each pair's agreement is measured after removing the gradient directions explained by all other tasks. The computation costs one inversion of a small tasks-by-tasks matrix per logging interval. The claim is that grouping tasks by this partial affinity predicts true multi-task gains better than grouping by raw pairwise affinity. The gap should be largest when many tasks share a dominant generic component. The note also makes a falsifiable prediction about where the method should not help: when tasks are trained on largely different examples.

Core Claim

Let $g_i^{(t)}$ be the gradient of task $i$'s loss with respect to the shared parameters at logged step $t$, for $T$ tasks. Accumulate the Gram matrix $M = \sum_t G^{(t)}$, where $G^{(t)}_{ij} = g_i^{(t)} \cdot g_j^{(t)}$. Then define

$$P = (M + \lambda I)^{-1}, \qquad a_{ij} = -\frac{P_{ij}}{\sqrt{P_{ii} P_{jj}}},$$

where $\lambda$ is a small ridge term and $a_{ij}$ is the partial affinity between tasks $i$ and $j$. This is the precision-matrix identity for partial correlation, applied with gradient coordinates as observations and tasks as variables. Algebraically, $a_{ij}$ is the cosine between $g_i$ and $g_j$ after each is projected off the span of the remaining $T-2$ task gradients. Pairwise measures such as gradient cosine, first-order lookahead (Yang et al.) and TAG's lookahead loss change (Fifty et al.) credit a pair with agreement that is really agreement with a third task. $a_{ij}$ does not.

We keep everything downstream fixed, including the grouping solver and the training recipe. We swap only the affinity matrix, and ask whether groups chosen from $a_{ij}$ train to lower loss.

Relation to Prior Work

The motivation comes from two places. Ernst et al. show that on SummEval, correlations between automatic metrics and human coherence judgments largely vanish once relevance is held fixed, with a median relative difference of 0.82. Their finding shows that apparent agreement across several criteria can be carried by one shared driver. We carry that diagnosis from evaluation metrics to task gradients. Lin et al. similarly use partial correlation to isolate generation–judgment links while controlling for answer correctness. The ecology studies (K. Wang et al., T. Wang et al.) use the same tool to separate co-varying climate drivers. They show it is routine elsewhere, but they supply no evidence about neural networks.

On the multi-task side, several methods estimate affinity and then solve a grouping problem. TAG (Fifty et al.), the Taylor approximation of Yang et al., ETAP (ETAP), which corrects pairwise gradient scores with splines and ridge-regression residuals, and the mixed-integer grouping of C. Wang et al. all score pairs one at a time. None of them conditions a pair's score on the other tasks. ETAP's residual correction is the closest in spirit, but it residualizes predicted gains against observed gains, not tasks against tasks. DMTG skips explicit affinity by learning group assignments with a Gumbel-Softmax relaxation. It is a competitor on outcomes, not a close relative in method.

Gradient surgery methods also act on pairs. PCGrad (Yu et al.) projects away pairwise conflicts. CAGrad (Liu et al.) solves jointly over all gradients for an update. RotoGrad (RotoGrad) rotates task representations. These methods change the update rather than deciding which tasks share a network, so they compete with grouping rather than supplying its affinity.

Work on spurious cross-task shortcuts addresses a different object. Chai et al. detect shortcuts from shifts in pairwise label correlation under resampling. MT-CRL decorrelates expert modules with Pearson penalties plus environment-based invariance. Both work on labels or representations; neither conditions gradient affinity on other tasks.

Three prior results constrain the design. First, Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation shows that gradient similarity tracks real task relationships only when tasks share at least about 40% of training examples, with a transition near 29.7%. Partialling removes shared structure among tasks; it cannot restore information that is absent because tasks never saw the same inputs. We therefore predict no rescue below that threshold and test for it. Second, the graphical-model literature warns that conditioning on a collider manufactures dependence. Here a collider would be a task whose target combines two others' targets, and it could create false partial affinity between those two. Third, when $T$ approaches the number of informative gradient directions, the inversion becomes unstable. For $T=40$ on CelebA we therefore use a sparse regularized precision estimate, as in graphical lasso. The learned sparsity of uGLAD is an option, although uGLAD itself used a multi-task setup only as a consensus trick across missing-data batches.

Research Program

The work has three parts.

Controlled test. We build synthetic multi-task regression and classification problems with known structure. A tunable generic factor drives all tasks. A planted sparse set of direct task pairs shares private factors. Some configurations include a planted collider task. We then check whether $a_{ij}$ recovers the planted direct pairs, measured by adjusted Rand index and area under the curve against the planted graph. We also map how recovery degrades as the generic factor's strength and the collider's presence vary.

Grouping on real benchmarks. On real benchmarks we log sketched gradients at fixed intervals. Random projection to 4,096 dimensions preserves inner products approximately. We compute $a_{ij}$ with $\lambda$ set by a held-out criterion. We then feed $a_{ij}$ into TAG's group-selection procedure and, separately, into the mixed-integer solver of C. Wang et al., in place of their native scores.

Diagnostics. We decompose each raw pairwise score into its direct part and the part explained by the other tasks. Where grouping improves, this shows which third tasks were confounding which pairs.

Baselines

The naive reference points are training all tasks in one shared network and training each task alone. DMTG reports a 22% total-loss reduction over the shared network and 13% over single-task training on Taskonomy-5 (DMTG). These figures anchor how much headroom grouping has.

The strongest affinity baseline is ETAP. It is the natural competitor because it also post-corrects pairwise gradient affinities. On ETTm1 it reports a Pearson correlation of 0.84 between predicted and true pairwise gains, against 0.47 for TAG (ETAP). We will rerun ETAP and TAG ourselves with the same backbone, data splits, gradient-logging schedule and number of logged steps, so that each affinity method gets the same compute. Beating ETAP means exceeding its correlation with ground-truth gains on ETTm1, CelebA-9 and Taskonomy-5, with confidence intervals from three seeds. It also means giving lower held-out total loss for the chosen groups at equal numbers of groups.

DMTG is the strongest end-to-end grouping competitor and is run at the same total GPU budget. For reference, DMTG reports about 1 hour on CelebA-9, against more than 18 hours for higher-order approximation. As a non-grouping competitor we run CAGrad on the single shared network. CAGrad reports a 0.20% average per-task drop on NYUv2 against 3.97% for PCGrad (Liu et al.). No reported result was found on Lacuna for CAGrad on CelebA-9, Taskonomy-5 or ETTm1 grouping settings.

Two ablations isolate the key ingredient. The first computes the raw cosine from the very same accumulated $M$, so partialling is the only difference. The second removes only the consensus gradient $g_0$, the average of the task gradients, before computing cosines; it tests whether a single shared direction explains the effect.

Why not just…?

Why not just subtract the average gradient before computing cosines? This is the most serious threat, because the confounder we describe might be a single dominant generic direction. If so, removing $g_0$ captures nearly all the benefit at the cost of one projection. Full partialling differs in two ways. It removes a subspace rather than one direction, and it removes directions attributable to specific third tasks rather than to their average. The consensus-only ablation settles the question. If it matches full partialling on all three benchmarks, we will report that the cheap version suffices and recommend it. The synthetic study, where the generic structure can be made one-dimensional or multi-dimensional, predicts where the two should diverge.

Why not just use TAG's lookahead, which measures transfer directly? Lookahead measures the change in task $j$'s loss after a step on task $i$. That step also improves the generic features every task needs, so lookahead inherits the same shared-driver confound. ETAP's 0.84 against TAG's 0.47 on ETTm1 already shows that raw lookahead leaves recoverable signal on the table. Our comparison asks whether conditioning recovers more of it than ETAP's residual correction does.

Why not skip grouping and train one network with CAGrad? If a joint conflict-averse update removes interference, grouping is unnecessary. The same-budget comparison against CAGrad on the single shared network answers this directly. We treat a CAGrad win on a benchmark as evidence that grouping, by any affinity, is the wrong tool there.

Why not learn the grouping end to end, as DMTG does? DMTG avoids affinity estimates altogether and is fast. It offers no explanation of which tasks confound which, and its relaxation is a separate source of error. If DMTG matches partial-affinity grouping on held-out loss at equal budget, our contribution narrows to the diagnostic decomposition. We state that outcome as acceptable rather than argue around it.

Proposed Experiments

Recovery of planted structure. The first decisive check is the synthetic benchmark. $a_{ij}$ should recover the planted direct pairs with clearly higher adjusted Rand index than raw cosine as the generic factor strengthens. A planted collider should produce a false edge, which we then try to suppress by excluding tasks flagged as colliders from the conditioning set.

Agreement with ground-truth gains. The second check uses measured gains. We train every task pair on CelebA-9 (36 pairs), ETTm1 (21 pairs) and Taskonomy-5 (10 pairs), plus all 31 subsets of Taskonomy-5. We then compare how well each affinity predicts pairwise and group gains, and how the selected groups perform on held-out data.

NYUv2. With only three tasks, NYUv2 is the setting where partialling should matter least. It serves as a sanity check that the method does no harm; there, conditioning each pair on the one remaining task is a full partial.

Low-overlap boundary. The third check reuses the molecular benchmarks of the information-theoretic study: Tox21, ToxCast and TDC ADME. We subsample shared examples to control overlap. Our prediction is that partial affinity improves over raw similarity above about 40% overlap and fails equally below the transition. If it instead helped below the threshold, that would contradict our own account of why it works, and we would report that.