Nonparametric Weight-Factor Allocation for Joint Multi-Task Training

When one neural network is trained on many tasks at once, the tasks often pull its shared weights in opposite directions, and some tasks end up worse than if they had been trained alone. Today's fixes take one of two forms. The first edits the gradients after the fact, projecting away the parts that conflict. The second hard-codes which layers are shared and which belong to each task. We propose a third option. Each layer's weight matrix is built from a dictionary of simple rank-1 pieces, called "factors". Each task chooses, through a learned on/off switch per factor, which pieces it uses. The number of pieces is not fixed in advance: a Bayesian nonparametric prior, the Indian Buffet Process, lets the dictionary grow only as fast as task diversity demands. Tasks that agree end up sharing factors. Tasks that conflict end up on disjoint factors, so their gradients never meet.

The core claim is testable. Learned factor allocation, trained jointly on all tasks and never frozen, should do two things. It should lower the measured gradient conflict on the parameters that tasks actually share. On diverse benchmarks such as the 50-task Meta-World suite, it should match or beat conflict-averse gradient surgery while needing one backward pass per step. And because the dictionary size is inferred, the same hyperparameters should carry over from 10 tasks to 50 without retuning. That last property is what a deterministic sparse-gating model with a fixed number of factors cannot offer.

Where This Sits

Three prior lines meet here, and each leaves a specific hole.

The first line motivates the approach but works only in shallow models. Factorial Multi-Task Learning clusters tasks with a Dirichlet process and lets clusters partially share basis vectors through a hierarchical Beta process. It reports a London Schools RMSE of 9.73 against 10.67 for single-task learning. However, it uses Gibbs sampling over linear and logistic models on fixed features. Our proposal keeps its idea of partial relatedness but moves it inside the weight matrices of a deep network and replaces sampling with stochastic variational training.

The second line supplies our parameterization, in a different regime. Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors (IBP-WF) writes each task's weights as an IBP-gated sum of rank-1 factors and trains with variational inference. It handles tasks in sequence, and it freezes every factor once that factor's selection probability crosses a threshold, so that old tasks are not forgotten. We reuse the parameterization almost directly. The difference is that we train all tasks simultaneously and never freeze. Every factor keeps receiving gradients from every task that selects it.

Two related papers show that shared on/off selection with separate continuous weights can carry correspondence across domains or classes. BP-JDL does this across resolutions, and JBDC does it across classes. Both are shallow and single-objective.

The third line defines the competition. PCGrad projects each task's gradient off any other task gradient it conflicts with. CAGrad maximizes the worst-case task improvement within a ball around the average gradient. RotoGrad learns per-task rotations of the shared features. All three treat the shared parameters as one undivided block that every task touches, and all need per-task gradients. Our bet is that much of the conflict they repair would not arise if tasks did not share parameters they disagree about.

There is also a constraint on design. Online Multi-Task Learning via Sparse Dictionary Optimization shows that a fixed-size task dictionary (ELLA-SVD) degrades when tasks have heterogeneous input distributions. This is one reason we do not fix the dictionary size.

Core Mechanism

For a layer with base weight $W_0$, task $t$ uses

$$W_t = W_0 + \sum_{k=1}^{K_{\max}} z_{tk}\, s_{tk}\, u_k v_k^\top,$$

where the terms are defined as follows. The vectors $u_k, v_k$ form the $k$-th shared rank-1 factor. The scalar $s_{tk}$ is task $t$'s weight on that factor. The variable $z_{tk}\in\{0,1\}$ is task $t$'s on/off switch for it. The bound $K_{\max}$ is a generous truncation of the infinite dictionary.

The switches follow a stick-breaking Indian Buffet Process prior: $z_{tk}\sim\text{Bernoulli}(\pi_k)$ with $\pi_k=\prod_{j\le k}\nu_j$ and $\nu_j\sim\text{Beta}(\alpha,1)$. The parameter $\alpha$ controls how fast new factors are opened. We train a variational posterior over $z$ using a relaxed-Bernoulli (concrete) reparameterization, as IBP-WF does. The objective is the sum of task losses plus the KL term to the prior, and all tasks contribute to every minibatch.

Because a factor only receives gradient from tasks whose switch is on, the gradient on factor $k$ is a sum over the set $\mathcal{T}_k$ of tasks selecting it, rather than over all $T$ tasks. We define per-factor conflict as the mean pairwise cosine between task gradients on $(u_k,v_k)$ within $\mathcal{T}_k$. This quantity is the main diagnostic.

We also build an optional variant that applies PCGrad-style projection only within each factor among the tasks in $\mathcal{T}_k$. Its pairwise cost scales with $\sum_k |\mathcal{T}_k|^2$ on small vectors, rather than $T^2$ on the full parameter vector. It still needs per-task gradients, so it is a diagnostic variant and not the headline method. The headline method uses one backward pass per step.

Research Program

The work proceeds in three stages.

Stage 1: verify inference in the shallow setting. We first check the inference recipe where ground truth exists. On synthetic multi-task regression with planted task groups, and on London Schools, we run a linear version of the model. We ask whether it recovers the planted sharing structure and approaches the F-MTL result without Gibbs sampling. If stochastic variational training cannot match F-MTL here, the deep version has no foundation.

Stage 2: build the deep version. We implement the factorization in the shared layers of the actor and critic networks of multi-task Soft Actor-Critic (SAC), a standard off-policy reinforcement learning algorithm, for Meta-World. We do the same in the shared encoder of an MTAN-style network for NYUv2 dense prediction.

Stage 3: analysis. We track per-factor conflict over training and compare it with full-parameter conflict under ordinary joint training. We examine whether the switch pattern clusters tasks in interpretable ways, for example by grouping Meta-World skills by shared motion primitives. We measure how the number of active factors grows from MT10 to MT50 under fixed $\alpha$. The IBP predicts sublinear growth, because the expected number of instantiated features grows harmonically in the number of tasks.

Baselines

The naive reference points are two. The first is plain equal-weighted joint training on a shared backbone: multi-task SAC on Meta-World and MTAN on NYUv2. The second is independent single-task models, which define the negative-transfer line. The Meta-World paper reports multi-headed SAC at 85% success on MT10 and 40% on MT50. This shows that diversity, not task count alone, is what breaks shared models.

The strongest existing competitor is CAGrad. It directly targets the failure we address, it is the best reported method in the file on both of our benchmarks, and its fast variant scales to 50 tasks. CAGrad reports an MT50 success rate of 0.52, against 0.50 for PCGrad and 0.36 for multi-task SAC. On NYUv2 it reports a per-task performance drop of 0.20%, against 3.97% for PCGrad and 5.59% for MTAN. The PCGrad paper itself reports about 70% on MT50 under its own protocol. Because the protocols evidently differ, we rerun every method in one codebase. All methods get the same environment steps, the same network width plus a parameter-matched adjustment for the factor dictionary, and the same hyperparameter-search budget per method, over five seeds. Beating CAGrad means a higher MT50 success rate with non-overlapping seed intervals, or matching it at clearly lower wall-clock cost. On NYUv2 it means a lower $\Delta m$ than CAGrad's rerun value. We also report RotoGrad and PCGrad on NYUv2.

Two ablations isolate the proposal's ingredients. The first is a deterministic gated dictionary: the same rank-1 factors and the same $K_{\max}$, with learned sigmoid gates and an L1 penalty instead of the IBP prior. It is tuned separately on MT10, and the tuned penalty is then reused unchanged on MT50, exactly as $\alpha$ is reused. This tests whether the nonparametric prior matters at all. The second is an IBP-WF-style run with sequential task order and factor freezing. This tests whether simultaneous training with co-adapting factors is actually the source of any gain. A third comparison pits allocation with per-factor surgery against allocation without it. This shows whether structural allocation has already removed the conflict that surgery would fix.

Why not just…?

Why not just use CAGrad or PCGrad on a standard backbone? This is the real threat, and we treat it as the main comparison rather than arguing around it. The mechanistic case against it is that surgery repairs conflict on parameters every task touches, while allocation can prevent that conflict. Still, CAGrad's 0.52 on MT50 is a strong bar. The per-factor conflict measurement settles the question. If allocated factors show near-zero conflict and allocation alone matches CAGrad, then surgery is unnecessary. If conflict on shared factors stays high and the surgery variant is needed to win, then the contribution narrows to cheaper, localized surgery, and we will report it that way.

Why not just give each task its own fixed-rank low-rank adapter, or use deterministic sparse gates? Per-task adapters with a fixed rank cannot share pieces across tasks, and they force one rank on all tasks. Deterministic gates over a shared dictionary can share, but they replace the dictionary-size problem with a sparsity-penalty problem. F-MTL's gap over fixed-rank GO-MTL (6.99% versus 13.4% MNIST error) suggests that rigid shared subspaces hurt under outlier tasks, but that evidence is shallow. The gated-dictionary ablation with its penalty transferred from MT10 to MT50 is the honest test. If it matches the IBP version, the Bayesian prior is not earning its keep, and we would recommend the simpler model.

Why not just train separate networks, or cluster tasks first and train one network per cluster? Separate networks eliminate conflict but also eliminate transfer and multiply parameters. Hard clustering is the "disjoint groups" extreme that F-MTL argues against, because tasks in different clusters can still share low-level structure. The single-task baseline and a cluster-then-train baseline, using clusters taken from our own learned switch patterns, show whether partial factor-level sharing beats hard partitions at equal parameter count.

Proposed Experiments

The decisive experiment is Meta-World MT10 and MT50 under one shared protocol. The model and the gated ablation use hyperparameters fixed after tuning on MT10. CAGrad, PCGrad, multi-task SAC, and single-task SAC are rerun alongside them. We report success rate, wall-clock time per step, and the number of active factors.

NYUv2 serves as the second domain. Data come from Silberman et al., with segmentation, depth, and normals scored by $\Delta m$ relative to single-task models. This tests whether the effect holds for dense vision tasks that are known to be closely related, where we would expect heavy factor sharing and a smaller gain.

London Schools and synthetic grouped tasks serve as the inference sanity check against F-MTL's reported RMSE.

Across all three settings, the diagnostic that would most change our minds is the per-factor gradient conflict. If learned allocation does not lower conflict on shared factors relative to full-backbone joint training, the central mechanism is wrong, whatever the headline numbers say.