Sampling Which Tasks Share Which Modules in Multi-Task Networks

When one neural network is trained on several tasks at once (multi-task learning), some tasks help each other and some quietly damage each other. The damage is called negative transfer. Current systems decide what to share in one of two ways. Some make a single learned choice of which shared building block each task uses. Others leave everything shared and then patch the gradients. Neither can say how confident it is that sharing a given block is a good idea.

We propose to treat "does task k use shared module m at layer l, or its own private copy?" as an uncertain quantity. We give it a prior and draw samples from its posterior with Markov chain Monte Carlo (MCMC), a family of algorithms that produce samples from a probability distribution by running a random walk whose long-run frequencies match that distribution. The samples cover this discrete choice together with the network weights. Put in two sentences: the network keeps a bank of reusable blocks, and a sampler repeatedly re-decides which task uses which block, based on how well each task's data is explained under each choice. The fraction of samples in which two tasks share a block becomes a direct, validation-free estimate of whether sharing helps. That estimate should route a harmful auxiliary task away from the modules the main task depends on.

Core Claim

Negative transfer is better handled as a model-selection question inside the posterior than as a gradient-geometry problem. The case rests on a disagreement in the literature. CAGrad and RotoGrad attribute negative transfer to conflicting and imbalanced task gradients. CAGrad bounds the update inside a ball around the average gradient; RotoGrad rotates shared features to align task gradients. ForkMerge argues the opposite. It shows that conflicting gradients can be harmless (weight decay produces them too) and that the real cause is auxiliary data pulling parameters away from the target optimum. It fixes this by forking the model and merging with a coefficient searched on a clean validation set.

ForkMerge's evidence motivates this proposal. Its dependence on a held-out split and a grid search is exactly what a posterior over sharing decisions replaces. If an auxiliary task's data distribution is incompatible with the target task's, the likelihood of the target data drops when the two share a block. The sampler then assigns low probability to that sharing, using only training data and a complexity-penalizing prior.

What would be new is posterior sampling over discrete module assignments in a deep multi-task network, jointly with the weights. Existing modular multi-task systems all make point decisions:

  • Compositional Models mixes a bank of K template blocks with task-specific coefficients, pushed toward one-hot choices by a clustering regularizer, all trained by gradient descent.
  • MUiR finds a block-to-hypermodule map with a (1+λ) evolutionary search.
  • AutoMTL chooses per-operator share, copy, or skip with a Gumbel-Softmax relaxation.

The Bayesian work in the file stops short of this. BayesNAS uses uncertainty to prune architectures, but through a local Laplace approximation on a single task. Sparse Bayesian Multi-Task Learning models task correlations with variational EM in linear models only. JaxSGMC supplies the continuous-weight sampler we reuse (stochastic-gradient Langevin and Hamiltonian samplers built from swappable parts), but it has only been tested on single-task problems.

Mechanism

We build on the isometric backbone of Compositional Models, where every block has the same input and output shape, so any block can sit at any layer. Each layer l of each task k carries an assignment $z_{k,l}$. Its value is either one of M shared template blocks or a private block for that task.

The continuous weights $\theta$ are updated with stochastic-gradient Hamiltonian Monte Carlo from JaxSGMC's components. That is a sampler that uses minibatch gradients plus injected noise and friction so that it wanders through the posterior instead of settling at one point.

Every few hundred weight steps, a discrete Gibbs step resamples each assignment from

$$p(z_{k,l}=m \mid \text{rest}) \propto \pi(m \mid z_{-k,l}) \, \exp\!\big(-\beta N_k \hat{L}_k(\theta; z_{k,l}=m)\big),$$

where the symbols are:

  • $\hat{L}_k$ is task k's average loss on a minibatch with the candidate block swapped in.
  • $N_k$ is task k's dataset size.
  • $\beta$ is a temperature that rescales the likelihood.
  • $\pi$ is a Chinese-restaurant-style prior. Its probability of reusing block m grows with the number of other tasks already using m, and it charges a fixed cost for a private block.

This is cheap: it needs M+1 forward passes on one minibatch per task and layer, and no backward pass.

The known risk is poor mixing. Once weights adapt to one assignment, other assignments look worse than they would after adaptation. This is the mode-trapping problem the MCMC literature flags for deep models. We address it in two ways. First, the likelihood is tempered at the start of training, so assignments can move before the weights commit. Second, several short chains run in parallel. Diagnosing mixing is itself part of the research program, not an afterthought.

The modular-design direction enters as a constraint rather than the headline. The assignment sampler must be a drop-in component alongside the architecture and weighting modules of LibMTL, with the weight sampler composed from JaxSGMC-style blocks. That way every deterministic baseline and our method run through the same trainer and data pipeline. This choice is also how we keep the comparisons below honest.

Research Program

The work has three stages.

The first stage builds the sampler. We then validate it on a synthetic multi-task regression in the style of Sparse Bayesian Multi-Task Learning's 5-task controlled setup. We extend that setup to small nonlinear networks with a planted sharing structure. Here exact enumeration or long full-batch Hamiltonian Monte Carlo gives a reference posterior over assignments, which lets us measure how much the fast sampler's sharing probabilities deviate from the reference.

The second stage tests the core claim with deliberately harmful auxiliary tasks. We corrupt or distribution-shift some auxiliary tasks by a known amount. We then check whether the posterior sharing probability between each auxiliary task and the main task falls as the corruption grows. For every task pair we obtain an "oracle" label for harmful sharing by training pairwise shared and unshared models and comparing main-task error. The sampler's sharing probabilities are scored against these labels.

The third stage scales to standard dense-prediction and multi-domain benchmarks. There we ask whether better sharing decisions translate into accuracy and calibration.

Baselines

The naive reference point is independent single-task training of the same backbone. It is the denominator of the standard average relative performance drop metric, Δm%. We also run hard parameter sharing with equal loss weights, the "just add the losses" approach. ForkMerge reports that standard multi-task methods fall 10–15% below single-task baselines on DomainNet, so this is where negative transfer is visible.

The strongest optimizer-side competitor on NYUv2 is CAGrad. It reports Δm% of 0.20% on NYUv2, against 5.59% for the MTAN architecture and 3.97% for PCGrad.

The strongest competitor on the auxiliary-task question itself is ForkMerge. It reports +2.00% over single-task on DomainNet, 4.03% relative improvement on NYUv2, and 1.30% on a 100-million-record AliExpress dataset. ForkMerge is the method most directly aimed at harmful auxiliary tasks, which is why it is the one to beat.

The strongest modular competitor is the deterministic router of Compositional Models, run on the identical template bank. We also run an ensemble of five such deterministic routers trained from different seeds, which is the main threat described below. On Taskonomy we include AutoMTL's Gumbel-Softmax search. It reports gains of 13.6% over DEN and 3.1% over AdaShare there; the file gives no absolute Δm% for AutoMTL on NYUv2 or Cityscapes. For the file's figures on SG-MCMC in a deep network, we note that JaxSGMC reports 57.32% CIFAR-10 test accuracy for its preconditioned Langevin sampler on MobileNetV1.

Two ablations isolate the key ingredient. The first samples weights with SG-MCMC but freezes the assignments at the deterministic router's solution; it tests whether sampling the discrete sharing structure matters. The second samples assignments but keeps point-estimate weights trained with SGD; it tests whether weight uncertainty matters.

All methods run in the same LibMTL-style trainer on the same splits and backbone. Each method gets the same number of gradient evaluations. For sampled or ensembled methods this is total compute across chains or members. Each method also gets the same hyperparameter search budget, and ForkMerge alone is allowed its validation split. We do not use the numbers above as our comparison points; we re-run everything under the same protocol.

Beating the baselines means three things together. On NYUv2, Δm% must be at or below CAGrad's rerun value. On DomainNet, the gain over single-task must match or exceed ForkMerge's rerun value without using ForkMerge's validation split. On the synthetic and corrupted-auxiliary studies, sharing decisions must be measurably better than the router ensemble's.

Why not just…?

Why not just ensemble several deterministic routers? This is the real threat. Five Compositional-Models routers trained from different seeds also produce an empirical sharing frequency and improve calibration. If they match our method at equal compute, the MCMC part adds nothing. The difference we expect is that seed variation reflects optimization noise, not how much the data supports each choice. So an ensemble may agree unanimously on sharing a harmful auxiliary block when the optimizer's inductive bias favors it. The corrupted-auxiliary study settles this directly: we score both methods' sharing probabilities against the oracle pairwise labels. If the ensemble scores as well, we will report that the benefit comes from ensembling, not from Bayesian treatment.

Why not just use Gumbel-Softmax search with a temperature, as AutoMTL does? Gumbel-Softmax relaxes the discrete choice so it can be optimized by gradient descent, and it converges to one architecture. Its sampling noise is a relaxation device, not a posterior, so it carries no evidence-versus-complexity trade-off. It also inherits the gradient-conflict framing that ForkMerge questions. We include it on Taskonomy under the same budget. If its final choices agree with our high-probability assignments and reach equal Δm%, the discrete posterior buys calibration at most, not better structure.

Why not just use ForkMerge? ForkMerge works and is the accuracy target. But it decides with a validation grid search over interpolation between whole models, and it assumes a clean validation split. It cannot make different decisions per layer or per task pair. We test the setting where this matters by shrinking or contaminating the validation split. If ForkMerge still matches us there, our per-module decisions are not buying anything.

Why not just use a Laplace or variational approximation, as in BayesNAS? A local Gaussian approximation around one assignment cannot represent a posterior split between "share" and "private," which is exactly the quantity we need. BayesNAS also caches activations for its Hessian, which limits its scale. The synthetic study with an exact reference posterior tests this claim. If a variational distribution over assignments recovers the reference as well as our sampler does, we would adopt it for its lower cost.

Decisive Checks

The idea fails outright under either of two outcomes. The first is that, in the controlled corrupted-auxiliary study, posterior sharing probability does not decrease as corruption grows, or tracks the oracle labels no better than the deterministic router ensemble. The second is that multiple chains started from different assignments end up with different sharing probabilities. That would mean we are reporting initialization, not a posterior, and chains that fail this check will be flagged as such.

The idea succeeds if three results hold together: sharing decisions are well ranked against the oracle, DomainNet performance is at or above ForkMerge without its validation split, and NYUv2 Δm% is at or below CAGrad's. It should also report lower expected calibration error than the deterministic baselines. Stage-level compute cost relative to CAGrad will be reported alongside every accuracy number.