Mitigating Multi Task Interference with Sparse Parameter Selection

24 concepts18 papers1 proposal

Multi-Task Learning (MTL) promises a compelling efficiency gain: by training a single model on a diverse set of tasks, features learned for one task can theoretically enrich and accelerate the learning of another. However, in practice, this joint training often triggers negative transfer—a phenomenon where the performance of the multi-task model falls below that of separate, single-task models trained from scratch.

To understand why negative transfer happens, consider a standard, densely shared neural network. When a model updates its weights, gradients from fundamentally conflicting tasks (such as fine-grained visual classification versus broad geometric localization) flow into the exact same parameter space. These gradients actively overwrite and degrade each other's specialized representations. The shared weights are forced into a compromised, noisy average that serves no individual task well.

To resolve this dilemma, a powerful paradigm has emerged: sparsity-driven parameter selection. Instead of forcing all tasks to share all parameters globally, these methods use sparsity constraints to dynamically isolate or couple tasks. By ensuring that each task only interacts with a sparse subset of the model's capacity, the architecture transforms a zero-sum parameter conflict into a cooperative, structured geometry where distinct tasks occupy nearly orthogonal subspaces.


Sparse Latent Bases and Dictionaries

One foundational way to implement this intuition is to decompose task parameters into a shared dictionary of latent "basis tasks" or "atoms." Instead of sharing raw parameters, tasks share the underlying dictionary, but each task retains its own unique, sparse strategy for combining those pieces.

In Learning Task Grouping and Overlap in Multi-task Learning, Abhishek Kumar introduces the GO-MTL framework, which models task parameters as sparse linear combinations of latent basis tasks. Because the combination coefficients are strictly constrained to be sparse, tasks that share no commonalities are completely isolated from one another, creating an inherent safeguard against negative transfer. Similarly, Haitham Bou-Ammar et al. apply this to reinforcement learning in Online Multi-Task Learning for Policy Gradient Methods. They demonstrate that sparse task-specific coefficients allow a control agent to selectively pull relevant skills from a shared library without corrupting unrelated control policies.

Densely Shared MTL (Conflict):
Task A Gradient ───┐
                   ├───> [ Shared Parameters ] <─── Task B Gradient (Overwrites A)

Sparsity-Driven Selection (Isolation):
Task A ───> [ Sparse Coeffs A ] ───> [ Basis 1 ] ───┐
                                                    ├───> Specialized Output
Task B ───> [ Sparse Coeffs B ] ───> [ Basis 2 ] ───┘

This concept scales effectively to complex, non-linear representations and fine-grained classification. In Learning to Share Latent Tasks for Action Recognition, Qiang Zhou et al. apply sparsity constraints to the combination weights of shared motion primitives. By enforcing sparsity on the internal features of these latent tasks, the model avoids over-sharing broad information that could dilute discriminative performance.

Taking parameter decomposition a step further, Li Shen 0005 et al. propose an Adaptive Sharing framework in Adaptive Sharing for Image Classification. They use an $\ell_1$ penalty to isolate fine-grained, class-specific features from a shared base. This explicit parameter decomposition prevents unrelated data from corrupting the shared feature space. To generalize this behavior across highly diverse task distributions, Boyu Wang et al. develop a hierarchical structure in Generalized Dictionary for Multitask Learning with Boosting that pairs non-linear atoms with sparse task-specific coefficients. The sparsity constraint ensures that outlier tasks naturally isolate themselves, preventing forced global information sharing from degrading the broader system.


Task-Level Routing and Mixture-of-Experts

With the rise of deep architectures like Transformers, static dictionary methods have evolved into dynamic, task-level routing via Mixture-of-Experts (MoE) frameworks. Rather than routing individual tokens independently, these approaches route entire tasks to specialized parameter modules, ensuring that specialized expertise is preserved.

In Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts, Qinyuan Ye et al. demonstrate that limiting the active parameter space through sparse task-level routing prevents unrelated tasks from overwriting each other's weights. This structural constraint allows the model to spontaneously cluster tasks in ways that mirror human intuition. In the domain of imitation learning, Dafni Antotsiou et al. develop MAPS in Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division. Their framework constrains an adaptive selector to choose only a sparse subset of available "proto-policies," preserving specialized robotic expertise while still allowing for the sharing of general-purpose skills.

Sparsity can also be formulated probabilistically or optimized during transfer learning to filter out interference:

  • Probabilistic Mixing: To handle data scarcity, Jiayi Shen et al. introduce Variational Multi-Task Learning (VMTL) in Variational Multi-Task Learning with Gumbel-Softmax Priors. By utilizing Gumbel-Softmax priors to generate sparse mixing weights, the model filters out interference from unrelated domains, ensuring that knowledge is aggregated only across highly relevant tasks.
  • Targeted Expert Selection: When adapting large language models to new domains, Zhisheng Lin et al. propose PEMT in PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning. By introducing a specialized sparsity-inducing loss function, the model is forced to select only a limited number of highly relevant source experts, preventing irrelevant task signals from diluting the target domain adaptation.

Parameter Masks and Harmony Subspaces

Rather than routing to discrete expert modules, another approach applies sparse masks directly to a model's weight updates. This strategy carves out non-interfering pathways within a single, shared weight matrix.

In HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning, Shengchao Hu et al. meta-learn task-specific masks that restrict each task to roughly 20% of the total model weights. These sparse masks define "harmony subspaces" where gradients from dozens of different robotic tasks can coexist without destructive interference.

Task Weight Matrix Updates:
Task A Mask: [1, 0, 0, 1, 0]  ───>  [■  .  .  ■  .]  (Subspace A)
Task B Mask: [0, 1, 1, 0, 0]  ───>  [.  ■  ■  .  .]  (Subspace B)
Combined Matrix:                    [■  ■  ■  ■  .]  (No Gradient Conflict)

This geometric property is equally powerful when merging independently fine-tuned models post-training. In Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks, MohammadReza Davari et al. apply a "band-pass" sparse mask to task-specific weight updates before merging them. The authors show that increasing the sparsity of these updates promotes geometric orthogonality between the task vectors. This structural property minimizes the interference between updates, allowing a single merged model to scale smoothly across many disparate tasks without retraining.


Uncovering Latent Task Relationships

Sparsity constraints do not just protect parameters; they also serve as an elegant mathematical tool to discover the underlying structure of the task space itself when relationships are unknown.

For instance, Yu Zhang et al. introduce the SPATS model in Learning Sparse Task Relations in Multi-Task Learning, enforcing $\ell_1$ regularization on task covariance matrices. This sparse covariance modeling acts as a selective filter, ensuring that a task's parameters are only influenced by data from statistically correlated tasks, thereby preventing noise transfer. Similarly, Carlo Ciliberto et al. propose Sparse Kernel Multi-Task Learning (SKMTL) in Learning multiple visual tasks while discovering their structure, using a non-parametric sparse approach to recover interpretable task graphs and decouple conflicting domains.

This structural discovery operates across several distinct methodologies:

By replacing dense, unconstrained parameter sharing with targeted, sparse allocation, multi-task systems resolve their internal conflicts. Sparsity provides the mathematical boundaries necessary to isolate unrelated tasks while allowing genuinely complementary domains to collaborate freely.

Key Foundational Work

Learning Task Grouping and Overlap in Multi-task Learning (2012) — contributed 1 concept elements to this direction.
Inferring latent task structure for Multitask Learning by Multiple Kernel Learning (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_negative_transfer_mitigation_selective_k_967805fa,
  title = {negative transfer mitigation (selective knowledge sharing)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/negative-transfer-mitigation-selective-knowledge-sharing/rdr_967805fa277e43e0861a23bd5901207c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_negative_transfer_mitigation_selective_k_967805fa_v1,
  title = {negative transfer mitigation (selective knowledge sharing)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/negative-transfer-mitigation-selective-knowledge-sharing/rdr_967805fa277e43e0861a23bd5901207c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake