Linear Weight Arithmetic for Modular Model Composition

155 concepts112 papers

When we fine-tune a neural network, we typically think of it as a complex, non-linear journey across a rugged loss landscape. However, recent discoveries reveal a surprising structural simplicity: if multiple models are fine-tuned from the same pre-trained checkpoint, the modifications made to their weights behave like vectors in a linear space. By subtracting the pre-trained base weights ($W_{\text{base}}$) from the fine-tuned weights ($W_{\text{ft}}$), we can isolate a task vector:

$$\tau = W_{\text{ft}} - W_{\text{base}}$$

This simple delta captures the precise parameter shift required to execute a specific capability. Because these models originate from a shared "weight basin"—a phenomenon explored by Yi-Lin Sung in An Empirical Study of Multimodal Model Merging—their internal representations remain aligned.

Why does this linear behavior emerge in highly non-linear models like Transformers? As shown by Zhanpeng Zhou et al. in On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm, fine-tuning under a shared initialization induces Cross-Task Linearity (CTL). This means that algebraic operations in the weight space map directly to predictable aggregations or subtractions of features in the representation space. Hongkang Li et al. provided the theoretical guarantees for this phenomenon in When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers, proving that task vectors accurately capture feature-learning dynamics even in non-convex landscapes.

Skill Composition via Vector Addition

Because task vectors behave linearly, we can combine distinct capabilities simply by adding their vectors to a base model:

$$W_{\text{composed}} = W_{\text{base}} + \tau_1 + \tau_2$$

This additive composition works because different tasks often occupy nearly orthogonal directions in the high-dimensional weight space. As noted in FusionBench: A Comprehensive Benchmark of Deep Model Fusion by Anke Tang, this relative orthogonality prevents different skills from interfering with or overwriting one another. For instance, Michael Luo et al. leverage this property in Stylus: Automatic Adapter Selection for Diffusion Models by summing adapters across distinct tasks to avoid visual artifacts and over-saturation.

This modularity enables training-free capability transfers across entirely different domains and modalities:

This additive nature also enables highly efficient training paradigms. In Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging, Jacob Morrison et al. introduce Parallel Train then Merge (PTM), where specialized "patches" are trained in isolation and added back to a general model, saving up to 95% of the computational cost of joint retraining.

Knowledge Negation via Vector Subtraction

If adding a task vector injects a behavior, then subtracting it should logically erase it. This operation forms the foundation of parameter-space machine unlearning:

$$W_{\text{unlearned}} = W_{\text{base}} - \tau$$

Subtracting a task vector acts as a surgical intervention to collapse specific undesirable pathways while leaving the surrounding general knowledge intact. This approach is widely applied across several domains:

Mitigating Feature Drift and Interference

While the ideal weight space is perfectly linear, real-world model merging faces practical challenges: combining too many vectors can cause "feature drift" or destructive interference where parameter updates conflict.

To handle these challenges, researchers use scaling factors ($\alpha$) to modulate the intensity of a task vector:

$$W_{\text{final}} = W_{\text{base}} + \alpha \tau$$

For example, Neale Ratzlaff et al. show in Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning that scaling and interpolating task vectors can perfectly re-balance a model's original text-reasoning capabilities with newly acquired multimodal skills. Similarly, in Sequential Editing for Lifelong Training of Speech Recognition Models, Devang Kulshreshtha et al. apply scaling factors to integrate new domain knowledge sequentially without displacing existing weights.

More sophisticated techniques move beyond global scaling to perform localized adjustments. Peng Ye introduces EMR-Merging in EMR-Merging: Tuning-Free High-Performance Model Merging, which uses binary masks to correct sign conflicts and scalar values to adjust the magnitude of unified task vectors. To minimize the discrepancy in internal representations between individual expert models and the merged model, Wenju Sun et al. propose layer-wise optimal task vector merging (LOT Merging) in Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration, ensuring that the internal feature distributions remain stable.

Key Foundational Work

Learn From Model Beyond Fine-Tuning: A Survey (2023) — contributed 2 concept elements to this direction.
Editing models with task arithmetic (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_task_vector_arithmetic_03914bce,
  title = {task vector arithmetic},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-vector-arithmetic/rdr_03914bcec99742569157e88cff2e17fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_task_vector_arithmetic_03914bce_v1,
  title = {task vector arithmetic},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-vector-arithmetic/rdr_03914bcec99742569157e88cff2e17fb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake