Granular Weight Merging for Multi-Task Foundation Models

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 144 papers62 conceptsfrozen 2026-10-04

The standard approach to creating a "Swiss Army Knife" model—one that excels at multiple specialized tasks—is to fine-tune a foundation model on each task separately and then average their weights. This technique, often called "Model Soups" or "Task Arithmetic," works surprisingly well when tasks are similar. However, as the diversity of tasks increases, simple global averaging begins to fail.

The core problem is destructive interference. When we apply a single global scalar to merge two models, we assume that every parameter in the network contributes equally to the task at hand. In reality, neural networks are hierarchical: shallow layers often capture general features, while deeper layers or specific submodules house highly specialized knowledge. Forcing a uniform merge across these different functional units leads to "parameter competition," where the specialized knowledge of one model overwrites the critical features of another.

To solve this, a new research direction has emerged: Granular Model Merging. Instead of using one weight for the whole model, researchers are developing methods to assign unique merging coefficients to layers, submodules, channels, and even individual parameters.

The Hierarchy of Granularity

The shift toward granularity happens across several architectural levels, each offering a different trade-off between control and computational complexity.

Layer-wise and Block-wise Merging

The most common form of granularity is layer-wise weighting. Enneng Yang and colleagues demonstrated in AdaMerging that assigning unique coefficients to each layer allows the model to preserve general features in shallow layers while specializing deeper ones. This is particularly effective in Federated Learning; Changlong Shi showed in FedAWA that calculating unique aggregation weights for every layer independently allows the global model to adapt to the different representational roles of various architectural components.

Submodule and Block Linearity

Moving deeper, researchers have found that while full Large Language Models (LLMs) may behave non-linearly, their internal components often retain a "local linearity" that makes merging easier. Rui Dai and others in Leveraging Submodule Linearity found that merging at the level of individual MLP blocks or Feed-Forward Networks significantly outperforms global task arithmetic. Similarly, when "recycling" Low-Rank Adapters (LoRAs) from the wild, Haokun Liu found that assigning distinct weights to specific attention or feed-forward matrices is crucial for selectively suppressing detrimental parameters within individual modules.

Channel and Parameter-level Precision

The highest resolution of merging occurs at the channel or parameter level. Ganggui Ding introduced Channel Merging, which clusters and merges parameters at the fine-grained channel level. By grouping individual output channels based on similarity, the system reduces the destructive interference that occurs when conflicting updates are forced into a single weight matrix. At the extreme end, Stefan Horoi and Guy Wolf proposed NP Merge, which learns a unique learnable interpolation weight for every individual parameter in the network, effectively bridging the performance gap between merged models and full ensembles.

Resolving Parameter Competition

Why does this granularity work? The intuition lies in Parameter Competition Balancing. As Guodong DU explains in PCB-Merging, different tasks often compete for the same weights. A parameter that is "critical" for a math task might be "redundant" for a coding task.

Granular merging acts as a high-resolution mask. By applying balancing coefficients at the individual parameter or channel level, the merging process can:

  1. Preserve task-critical weights where they are most needed.
  2. Suppress conflicting updates that would otherwise degrade performance.
  3. Align complex internal representations that a single global scalar would simply blur together.

Finding the Optimal Recipe

The challenge of granular merging is finding the right "recipe" of weights. Since the search space for parameter-wise or layer-wise weights is massive, researchers use several optimization strategies:

  • Evolutionary Search: Methods like LCSC by Enshu Liu use evolutionary algorithms to "blend" checkpoints, searching for the combination that maximizes generation quality.
  • Differentiable Merging: Differentiable Adaptive Merging (DAM) uses gradient descent to automatically learn scaling coefficients for weight matrix columns, making the process more efficient than brute-force search.
  • Zero-Order Optimization: For medical imaging, Guinan Su and colleagues developed MedSAMix, which uses zero-order optimization to discover layer-wise merging recipes without the need for full gradient descent.
  • Bayesian Uncertainty: Idan Achituve proposed BayesAgg-MTL, which treats gradients as probability distributions. By estimating the uncertainty of each gradient dimension, the model can prioritize "strict" dimensions where a task is highly sensitive while remaining flexible on others.

By moving from "mixing with a shovel" to "painting with a fine brush," granular weight merging allows us to recycle the vast library of specialized models into unified, multi-talented systems without the prohibitive cost of retraining from scratch.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_granular_weight_merging_for_multi_task_f_4e4e6cd2,
  title = {Granular Weight Merging for Multi-Task Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/granular-weight-merging-for-multi-task-foundation-models/rdr_4e4e6cd22adf4a8dad6dd821e86ea1fa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_granular_weight_merging_for_multi_task_f_4e4e6cd2_v1,
  title = {Granular Weight Merging for Multi-Task Foundation Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/granular-weight-merging-for-multi-task-foundation-models/rdr_4e4e6cd22adf4a8dad6dd821e86ea1fa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}