When several tasks train one shared network, their gradients often point in opposing directions. The network then settles on a compromise that serves no task well, and this is called negative transfer. The most popular fix, gradient surgery, works by subtraction. Before each update it removes the part of one task's gradient that conflicts with another task's gradient. The removed part is information the task needed, and it is simply discarded. We propose to keep it. Every weight matrix is written as a shared part plus a small task-private correction. The parts of each task's gradient that agree with the other tasks update the shared part. The parts that conflict, exactly what gradient surgery deletes, update that task's private correction. Conflict stops being a cost to minimize and becomes the training signal that decides what each task should keep to itself.
This joins two literatures that have grown up apart. Selective-sharing methods decide which parameters a task may touch, using masks, routing, or gradient projection. Additive weight decomposition, which writes a task's weights as a shared anchor plus a task-specific deviation, has worked in linear multi-task regression, continual learning, label-noise robustness, and model merging. In those settings, however, the split between shared and private is set by penalty strengths or by when tasks arrive, never by whether tasks currently agree. Our claim is that during simultaneous multi-task training, gradient agreement is the right signal for that split. Using it should make the shared backbone nearly conflict-free while recovering the task-specific accuracy that projection methods give up.
Core Claim
Consider a layer with shared weights $W$ and, for each task $t$, a task-private residual $\tau_t$ with the same shape but held sparse. Task $t$ uses $W + \tau_t$. At each step every task produces a gradient $g_t$ with respect to its effective weights. A conflict-resolution operator $P$ splits it into an agreeing part $P_t(g)$ and a remainder $r_t = g_t - P_t(g)$. In the PCGrad version, $P_t(g)$ is $g_t$ after projecting out its components along conflicting task gradients, so $r_t$ is exactly what PCGrad deletes. The updates are
$$W \leftarrow W - \eta \sum_t P_t(g), \qquad \tau_t \leftarrow \mathrm{Top}_k\big(\tau_t - \eta_\tau\, r_t\big),$$
where $\eta$ and $\eta_\tau$ are step sizes. $\mathrm{Top}_k$ keeps the $k$ largest-magnitude entries of each residual per layer, which caps how many private parameters each task can hold.
The shared update is identical to gradient surgery, so the shared backbone inherits its conflict-avoiding behavior. The residuals add capacity only where tasks actually disagree. When tasks agree, $r_t$ is zero and the model behaves like an ordinary shared network. The sparsity budget $k$ is the single capacity knob. The prediction is that a small budget, comparable to the 3.5–4% active-parameter fraction CALM found sufficient for merging, recovers most of the gap between joint training and single-task training.
Relation to Prior Work
Three groups of papers shape this proposal, each in a different role.
The first group motivates it. Gradient Surgery for Multi-Task Learning projects conflicting gradients onto each other's normal planes. Conflict-Averse Gradient Descent finds the update that best improves the worst task within a ball around the mean gradient. Both treat conflict as something to suppress inside one monolithic backbone, so no task ever gets to keep its disagreeing direction. Proactive Gradient Conflict Mitigation: A Sparse Training Perspective shows that structural constraints reduce conflict incidence and add to the gains of reactive methods. That is evidence that architecture and optimization address different parts of the problem. Its mask is fixed and shared by all tasks, whereas ours is task-specific and driven by conflict.
The second group supplies the decomposition. A Dirty Model for Multi-task Learning writes the linear parameter matrix as a shared block-sparse part plus an elementwise-sparse part, and shows the split helps when tasks overlap only partially. It reports 8.6% versus 9.9% error on handwritten digits. In that model the split is decided by penalty weights inside a convex problem. Ours is decided by gradient geometry in a deep network. Scalable and Order-robust Continual Learning with Additive Parameter Decomposition uses a masked shared base plus sparse task residuals, but tasks arrive one at a time and drift is corrected after the fact. We study simultaneous training, where conflicts happen concurrently. Federated Continual Learning with Weighted Inter-client Transfer uses a similar three-part split across clients, with attention-based retrieval. Tackling Noisy Labels with Network Parameter Additive Decomposition uses an additive sink to absorb harmful signal and then discards it. We retain the sink, because in multi-task learning the absorbed part is a task's specialized skill, not noise.
The third group constrains the design. HarmoDT learns task masks over a shared transformer through bi-level optimization based on agreement and importance scores. Its masks only remove parameters from a task and never add task-specific capacity, and the bi-level loop is expensive. Our routing is a single-level rule. Model Breadcrumbs shows that sparse task vectors with outliers trimmed interfere little when added together. That supports keeping residuals sparse, although Breadcrumbs works on models already trained separately, whereas ours shapes the residuals during training. MORRIS factorizes multi-task updates multiplicatively into a low-rank tensor and has no additive bypass for idiosyncratic directions.
Research Program
We will build the method in two versions. The exact version uses PCGrad's pairwise projection to compute $r_t$. Its cost grows with the square of the number of tasks, so it suits 3–10 tasks. The scalable version splits gradients coordinate by coordinate: an entry goes to the shared update when its sign agrees with the sign of the summed gradient, and to the task residual otherwise. This is close to HarmoDT's agreement score but needs no inner loop, and it is meant for Meta-World MT50.
Residuals are applied only to the top $L$ encoder blocks. This is a deliberate engineering constraint. In dense prediction a shared encoder runs once per image, and private residuals in every layer would force one encoder pass per task. Limiting them to late blocks keeps inference to one shared trunk plus $T$ short private branches. The number of residual blocks $L$ becomes a measured accuracy–cost tradeoff rather than a hidden cost.
The analysis has three parts. First, we will track conflict incidence in the shared weights over training to test whether routing actually yields a conflict-free shared base. Second, we will measure where residual mass ends up across layers and tasks. Third, we will check whether the overlap of residual supports between two tasks predicts the cost of training those two tasks jointly. If it does, the residuals double as a cheap measure of task relatedness.
Baselines
The naive reference points are plain joint training with equal loss weights and independently trained single-task networks. Following the convention of the dense-prediction literature, the relative drop $\Delta m\%$ is measured against the single-task networks.
The strongest existing competitor on dense prediction is CAGrad. On NYUv2, Conflict-Averse Gradient Descent reports $\Delta m\% = 0.20\%$, against 3.97% for PCGrad and 5.59% for MTAN. On Meta-World MT50 the same paper reports a success rate of 0.52 for CAGrad and 0.50 for PCGrad. The PCGrad paper itself reports roughly 70% on MT50 under its own protocol. That discrepancy is why we will re-run every baseline in one codebase rather than copy numbers. Sparse training with parameter selection per neuron is also included. Under joint training on NYUv2, Proactive Gradient Conflict Mitigation reports $\Delta m\%$ improving from 6.76% to 5.31%. We include it in combination with CAGrad as well, since that pairing is its strongest reported use. HarmoDT is the baseline for offline Meta-World. It reports a 57.2% success rate on 50-task sub-optimal data.
Fairness rules apply to all methods. They share the same backbone, data, number of epochs, and hyperparameter search budget, with equal trials per method. Every baseline is given the same extra parameter count our residuals use, in the form of wider task-specific heads, so that any gain cannot be attributed to capacity. Wall-clock time per step is reported for every method.
Two ablations isolate the key ingredient. The first is the obvious "dirty deep network": the same $W + \tau_t$ structure with the same top-$k$ budget, but with each $\tau_t$ trained on its full gradient $g_t$ and the shared part trained by plain summation or by CAGrad. The second replaces the conflict remainder $r_t$ with a random component of equal norm.
Beating CAGrad means a lower $\Delta m\%$ on both NYUv2 and CityScapes across three seeds, using a matched parameter budget and no more than about 1.5 times its step time. On MT50 it means a higher success rate than CAGrad and PCGrad run in our codebase.
Why not just…?
Why not just use CAGrad or PCGrad on a single backbone? These methods keep every shared weight agreeable by deleting each task's conflicting directions. A task whose optimum truly differs from the others therefore never reaches it. The dirty-model evidence that a monolithic shared model underperforms one with a private sparse component (8.6% vs. 9.9% error, A Dirty Model) is suggestive but comes from linear models. The direct test is our comparison against capacity-matched CAGrad. If the routing adds nothing once heads are widened, the claim fails.
Why not just add sparse task residuals and train everything with ordinary gradients? This is the most serious threat, and it is the first ablation. With full gradients, both $W$ and $\tau_t$ receive the same signal. The split between them is then arbitrary, set by step sizes and sparsity, and conflict still reaches $W$. Our hypothesis is that this variant leaves shared-weight conflict incidence close to that of plain joint training and lands between joint training and our method. If it matches our method, the honest conclusion is that additive capacity alone is what matters, and the routing rule should be dropped.
Why not just learn task masks the way HarmoDT does? Masks can only take shared parameters away from a task. They cannot give the task a direction that the shared weights lack. They also require bi-level optimization, which the file names as a major source of overhead. We compare against HarmoDT on its own offline Meta-World setting at matched training time. Its 57.2% is the number to beat.
Why not just train separate models and merge them? Merging methods such as CALM reach 87.7% against an 88.9% joint-training reference on 8-task ViT-B/32. That is still below joint training, and they require fully trained single-task models in the first place. Our method is trained in place and should be compared with joint training, not with merging.
Proposed Experiments
The primary testbeds are NYUv2 (segmentation, depth, surface normals) and CityScapes (segmentation, depth), evaluated with $\Delta m\%$ and per-task metrics using the architectures from the CAGrad and sparse-training papers. Meta-World MT10 and MT50 test scaling with the coordinate-wise version, measured by success rate. The offline 50-task setting allows a head-to-head comparison with HarmoDT.
Four checks decide the outcome. First, the full-gradient-residual ablation and the random-remainder ablation must both underperform the routed method; otherwise conflict routing is not the cause of any gain. Second, conflict incidence in the shared weights should fall below that of sparse training, which reports a 5.5–7% reduction. Third, accuracy should rise and then plateau as the residual budget $k$ sweeps from 0.5% to 10%, and $k = 0$ should reproduce PCGrad exactly as a sanity check. Fourth, on MT50, the gain over CAGrad should grow with the number of tasks, as expected if conflicting directions pile up across more task pairs. A flat or shrinking gain would suggest that sparse residuals become crowded at scale.
References
- CALM: Consensus-Aware Localized Merging for Multi-Task Learning
- Gradient Surgery for Multi-Task Learning
- Conflict-Averse Gradient Descent for Multi-task learning
- Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective
- A Dirty Model for Multi-task Learning
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
- Federated Continual Learning with Weighted Inter-client Transfer
- Tackling Noisy Labels With Network Parameter Additive Decomposition
- HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning
- Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks
- Low-Rank Updates of pre-trained Weights for Multi-Task Learning