Weighting Tasks by What They Cause: Counterfactual Credit for Multi-Task Loss Balancing

When one network is trained on many tasks at once, practitioners reweight each task's loss during training so that no task dominates. The most common rules watch how fast each task's loss is falling and give more weight to tasks that lag. That signal is confounded. A task's loss can fall because of its own gradient updates, or because another task's updates happened to improve the shared features, or it can rise because another task is damaging them. Current weighting rules cannot tell these apart. They credit or blame the task whose loss moved, not the task that moved it. We propose to separate the two using the core move of causal effect subtraction: estimate the effect of each task's updates on every other task's held-out loss, subtract those cross-task effects, and weight tasks by what remains. In linear causal models, subtracting a known direct effect from a variable yields an "auxiliary variable" that behaves as if the incoming edge had been cut.

The payoff is a weighting rule that acts on the cause of negative transfer rather than on its symptom. Negative transfer here means that joint training does worse on a task than training that task alone. When task A's updates hurt task B, a pace-based rule boosts B and a validation-based rule gives up on B. Neither touches A. A causal ledger identifies A as the source, reduces A's weight, and yields for B an estimate of how it would have trained without the others. This targets a documented failure: on a 128-task chemistry benchmark, the best loss-balancing rule still left 42 tasks worse than their single-task models.

Core Claim

The claim has two parts. First, a cheap online estimate of the direct effect of task $j$'s updates on task $i$'s held-out loss can be computed during training. Second, dynamic weighting driven by these subtracted effects reduces negative transfer beyond both pace-based and conflict-projection methods. The estimate is a running sum of first-order effects:

$$A_{ij}(t) = \sum_{s \le t} \eta_s\, w_j(s)\, \langle \nabla_\theta L_i^{\text{val}}(\theta_s),\; g_j(\theta_s) \rangle .$$

Here $\eta_s$ is the learning rate at step $s$, $w_j(s)$ is task $j$'s loss weight, $g_j$ is task $j$'s training gradient on the shared parameters $\theta$, and $L_i^{\text{val}}$ is task $i$'s loss on a held-out batch. A positive $A_{ij}$ is the estimated reduction of task $i$'s held-out loss caused by task $j$'s updates. This is an interventional quantity: to first order, it is what task $i$ would have lost had task $j$'s updates been removed. Task $i$'s counterfactual own-progress is then its observed held-out loss plus the effects contributed by others, $\tilde{L}_i(t) = L_i^{\text{val}}(t) + \sum_{j \ne i} A_{ij}(t)$. This mirrors the auxiliary variable $Y' = Y - \beta X$, with the sign set so that help from others is removed.

The weighting rule has two terms. The first sets task $i$'s pace from $\tilde{L}_i$ rather than from the raw loss, in the style of loss-ratio balancing. The second multiplies task $j$'s weight by a factor that shrinks when its net effect on others, $\sum_{i \ne j} A_{ij}$, has been persistently negative. The weight sum is renormalized as in GradNorm.

The key difference from existing methods is attribution. PCGrad and CAGrad react to the geometry of training gradients at each step. GradNorm, LBTW, DWA and CoBa react to each task's own loss curve. None of them assigns responsibility for one task's held-out change to another task.

Relation to Prior Work

Two papers supply the method. The auxiliary-variable papers (Incorporating Knowledge into Structural Equation Models Using Auxiliary Variables; Identification and Model Testing in Linear Structural Equation Models using Auxiliary Variables) show that subtracting a known direct effect removes an edge from a linear causal graph. Efficient Identification in Linear Structural Causal Models with Auxiliary Cutsets extends this with ACID, and its TED procedure breaks a total effect into a sequence of direct effects. We borrow the subtraction idea directly.

We borrow the decomposition idea for a subtle point. A dynamic weighting controller creates indirect paths of its own: task $j$'s update changes task $k$'s loss, which changes $w_k$, which changes task $k$'s updates, which changes task $i$. Our ledger measures only the direct edge. We use a TED-style unrolling over training steps as an analysis tool to check whether indirect paths through the controller are large enough to matter. The linear setting of those proofs is a real constraint, since neural networks are not linear. The file's neural tangent kernel analysis (When and why PINNs fail to train) motivates relying on local linearity over short windows, and we test that reliance directly rather than assume it.

Predictive Coding beyond Correlations shows that interventions can be simulated at runtime by clamping error signals, without rewiring the network. Its cut is all-or-nothing and it does not address multi-task training. Our ledger is a graded, continuous analogue, applied to task-to-task influence.

Three papers motivate the gap. Loss-Balanced Task Weighting reports that 42 of 128 PubChem tasks still suffer negative transfer under its best setting, and over 100 under GradNorm variants. The authors attribute this to fundamental incompatibility. We test the alternative explanation that the weighting signal was simply misattributed. Improving Multi-Task Generalization via Regularizing Spurious Correlation (MT-CRL) shows that shared encoders let tasks borrow each other's spurious features. It handles this by discrete routing plus an invariance penalty that needs labeled environments. Our method needs no environment labels and no second-order gradients, and it subtracts graded amounts rather than dropping pathways. CORE-MTL argues that gradient conflict is a symptom of entangled representations and fixes it architecturally, with physical rendering priors where available. That work is complementary. It changes the representation, whereas we change which task is credited and blamed, so the two can be stacked.

Two papers constrain the design. Gradient Surgery for Multi-Task Learning uses the word "surgery" for a projection that removes whatever component opposes another task's training gradient, with no causal reading. Conflict-Averse Gradient Descent shows that such projection can stall and reports a better alternative. Our ledger is built from held-out gradients of the victim task because conflicts between training gradients may be noise or harmless.

Research Program

We will build the ledger as a plug-in for existing multi-task training code. Every $K$ steps it computes the matrix of inner products between held-out gradients and training gradients. To keep this affordable at 40 to 128 tasks, it is restricted to the shared parameters and compressed with random-projection sketches of the gradients. This is a single batched matrix product, not PCGrad's sequential pairwise projections. The ledger is smoothed with an exponential moving average, because single-batch inner products are noisy. This noise is the practical counterpart of the "generic identifiability" caveat in the linear causal literature: small true effects will be hard to tell apart from zero.

The program has three stages. The first is validation. We check whether the ledger's predictions match real interventions: at chosen checkpoints, we branch training, drop task $j$ for a short window, and measure the actual change in task $i$'s held-out loss against $A_{ij}$ over that window. The second stage uses the ledger as a diagnostic, asking whether cumulative $A_{ij}$ early in training predicts which tasks end up with negative transfer against single-task models. The third stage uses the ledger as a controller, applying the weighting rule described above. We also test a pair-specific variant that removes the component of $g_j$ along $\nabla L_i^{\text{val}}$ only for pairs whose ledger entry has stayed negative, which we call causally gated projection.

Baselines

The naive reference is equal weighting ($w_i = 1$) with the learning rate tuned. The single-task models, trained with the same tuning budget, define negative transfer.

The strongest optimization competitor on dense vision is CAGrad (Conflict-Averse Gradient Descent), which picks the update that maximizes worst-case improvement near the average gradient. It reports a per-task drop $\Delta m\%$ of 0.20% on NYUv2, against 3.97% for PCGrad and 5.59% for MTAN. It is the right bar because it is the best reported conflict handler in the file. Beating it means a lower $\Delta m\%$ at matched backbone, epochs and tuning.

The strongest competitor on the negative-transfer question is LBTW (Loss-Balanced Task Weighting). It reports mean PR AUC of 0.253 on PubChem BioAssay and 42 negative-transfer tasks. The file's comparison points are 0.232 for single-task and 0.241 with 48 negative-transfer tasks for standard multi-task training. We count LBTW as beaten if the method reduces the negative-transfer count below LBTW's under our reproduction, while matching or exceeding its mean PR AUC.

We also run GradNorm (GradNorm), DWA (End-To-End Multi-Task Learning With Attention), CoBa (CoBa) and PCGrad. CoBa matters most because it already uses validation curves.

Every method gets the same architecture, data splits and epochs. Each gets an equal hyperparameter search budget of the same number of trials over its own knobs. All methods see the same held-out batches, so that access to validation data is not our method's private advantage. We report compute-matched results, with wall-clock time per epoch, since the ledger adds cost.

Three ablations isolate the key ingredient. The first uses only the pace term and drops the harm term. The second uses only the harm term and keeps raw-loss pace. The third builds the ledger from training gradients instead of held-out gradients. The third ablation is the direct test of whether causal attribution on held-out data, rather than gradient geometry, is doing the work.

Why not just…?

Why not just use CoBa, which already watches validation curves? CoBa lowers the weight of a task whose own validation loss starts to rise. When the rise is caused by another task, this abandons the victim and leaves the source untouched. The ledger lowers the weight of the source. This is a real threat if most negative transfer is self-inflicted overfitting rather than cross-task damage. In that case CoBa and our method should tie. The PubChem comparison of which tasks each method rescues settles the question, as does the intervention check of whether the ledger's large negative entries are real.

Why not just use PCGrad or CAGrad? They remove conflicts between training gradients step by step, without asking whether a conflict hurts anyone's held-out loss or which task is the source. CORE-MTL argues that geometry-only methods hit an error floor under distribution shift, although the file reports no number for that claim. Our third ablation, the training-gradient ledger, is exactly "attribution without held-out data". If it matches the full method, the causal framing adds nothing beyond geometry, and we will say so.

Why not just train single-task models once, find the harmful pairs, and regroup tasks? For 128 tasks this costs 128 extra training runs plus pairwise experiments, and the resulting grouping is static. The ledger's diagnostic stage tests whether early-training ledgers predict final negative transfer. If they do not, this cheaper static route may be the better tool, and the controller claim would weaken.

Why not just use MT-CRL's invariance routing? It needs labeled environments and second-order gradients (Improving Multi-Task Generalization via Regularizing Spurious Correlation). PubChem has no natural environment labels. We compare on NYUv2, where MT-CRL reports a 5.5% average gain over vanilla multi-task training across its suite, but no NYUv2-specific number is found in the file.

Proposed Experiments and Decisive Checks

PubChem BioAssay is the primary benchmark: 128 tasks, about 440,000 compounds, mean PR AUC, and a count of negative-transfer tasks against single-task models. NYUv2 (segmentation, depth and surface normals, reported as $\Delta m\%$) tests transfer to dense vision. CelebA's 40 tasks measure per-epoch cost against PCGrad, where the file reports only relative speedups for other methods. Meta-World MT50, where CAGrad reports 0.52 success, and MTBench (MTBench) are an optional extension. They are harder for this method because held-out losses in reinforcement learning are not stationary.

The first decisive check is intervention agreement. We measure the rank correlation between predicted $A_{ij}$ and the measured effect of actually dropping task $j$ for a window. If it is low beyond short windows, the first-order subtraction is not trustworthy and the controller results cannot be read causally.

The second check is diagnostic. We ask whether cumulative ledger harm after the first tenth of training ranks the eventual negative-transfer tasks on PubChem better than LBTW's loss ratio does.

The third check is the controller comparison, run against LBTW and CoBa on the negative-transfer count and against CAGrad on NYUv2 $\Delta m\%$, alongside the three ablations.

The fourth check is a TED-style unrolled analysis of how much of each total cross-task effect flows through the weighting controller itself. If that share is large, direct-effect subtraction is incomplete and the method needs the full decomposition.

A negative outcome would also be informative. Suppose the ledger agrees with interventions but downweighting the harmful source tasks does not reduce the negative-transfer count. That would support LBTW's reading that the residual tasks are genuinely incompatible rather than misattributed.