When we fine-tune a neural network, we typically think of it as a complex, non-linear journey across a rugged loss landscape. However, recent discoveries reveal a surprising structural simplicity: if multiple models are fine-tuned from the same pre-trained checkpoint, the modifications made to their weights behave like vectors in a linear space. By subtracting the pre-trained base weights ($W_{\text{base}}$) from the fine-tuned weights ($W_{\text{ft}}$), we can isolate a task vector:
$$\tau = W_{\text{ft}} - W_{\text{base}}$$
This simple delta captures the precise parameter shift required to execute a specific capability. Because these models originate from a shared "weight basin"—a phenomenon explored by Yi-Lin Sung in An Empirical Study of Multimodal Model Merging—their internal representations remain aligned.
Why does this linear behavior emerge in highly non-linear models like Transformers? As shown by Zhanpeng Zhou et al. in On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm, fine-tuning under a shared initialization induces Cross-Task Linearity (CTL). This means that algebraic operations in the weight space map directly to predictable aggregations or subtractions of features in the representation space. Hongkang Li et al. provided the theoretical guarantees for this phenomenon in When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers, proving that task vectors accurately capture feature-learning dynamics even in non-convex landscapes.
Skill Composition via Vector Addition
Because task vectors behave linearly, we can combine distinct capabilities simply by adding their vectors to a base model:
$$W_{\text{composed}} = W_{\text{base}} + \tau_1 + \tau_2$$
This additive composition works because different tasks often occupy nearly orthogonal directions in the high-dimensional weight space. As noted in FusionBench: A Comprehensive Benchmark of Deep Model Fusion by Anke Tang, this relative orthogonality prevents different skills from interfering with or overwriting one another. For instance, Michael Luo et al. leverage this property in Stylus: Automatic Adapter Selection for Diffusion Models by summing adapters across distinct tasks to avoid visual artifacts and over-saturation.
This modularity enables training-free capability transfers across entirely different domains and modalities:
- Cross-Modality Reasoning: Shiqi Chen et al. demonstrate in Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging that a reasoning task vector extracted from a text-only LLM can be added directly to a Vision-Language Model, boosting its problem-solving skills without further training. Similarly, Lingjie Jiang et al. combine vision and coding models via task vectors in VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models.
- Isolating Complex Capabilities: Complex behaviors like multi-step logic can be isolated. Mohammad Zbeeb et al. show in Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic that subtracting a baseline model from a reinforcement-learning-optimized model creates a "Reasoning Vector" that can be injected into other target models to instantly transfer chain-of-thought capabilities.
- Balancing Expertise and Safety: Instead of sequentially fine-tuning a model—which often causes catastrophic forgetting—researchers can fuse vectors in parallel. In Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs, Megh Thakkar et al. combine a domain-specific expertise vector with a safety-alignment vector onto a single base model, achieving an optimal trade-off without gradient-based optimization.
This additive nature also enables highly efficient training paradigms. In Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging, Jacob Morrison et al. introduce Parallel Train then Merge (PTM), where specialized "patches" are trained in isolation and added back to a general model, saving up to 95% of the computational cost of joint retraining.
Knowledge Negation via Vector Subtraction
If adding a task vector injects a behavior, then subtracting it should logically erase it. This operation forms the foundation of parameter-space machine unlearning:
$$W_{\text{unlearned}} = W_{\text{base}} - \tau$$
Subtracting a task vector acts as a surgical intervention to collapse specific undesirable pathways while leaving the surrounding general knowledge intact. This approach is widely applied across several domains:
- Copyright and Safety Enforcement: In Avoiding Copyright Infringement via Large Language Model Unlearning, Guangyao Dou et al. use vector subtraction to sequentially remove memorized copyrighted text. To remove harmful behaviors, Zheyuan Liu et al. propose Selective Knowledge negation Unlearning (SKU) in Towards Safer Large Language Models through Machine Unlearning, which intentionally trains a model on harmful patterns to isolate the exact "knowledge delta" before subtracting it.
- Distributional Correction: Subtraction can also eliminate systematic biases or data discrepancies. In Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognition, Hsuan Su et al. calculate the vector difference between models trained on synthetic versus real-world speech. Subtracting this synthetic-to-real delta from a new model significantly reduces its word error rate on real-world inputs.
- Feature Purification: Sometimes, task vectors contain redundant general language patterns that dilute their specificity. Mohammadtaha Bagherifard et al. demonstrate in GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction that subtracting a general-knowledge adapter from a task-specific adapter isolates a "residual" module with superior zero-shot generalization.
Mitigating Feature Drift and Interference
While the ideal weight space is perfectly linear, real-world model merging faces practical challenges: combining too many vectors can cause "feature drift" or destructive interference where parameter updates conflict.
To handle these challenges, researchers use scaling factors ($\alpha$) to modulate the intensity of a task vector:
$$W_{\text{final}} = W_{\text{base}} + \alpha \tau$$
For example, Neale Ratzlaff et al. show in Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning that scaling and interpolating task vectors can perfectly re-balance a model's original text-reasoning capabilities with newly acquired multimodal skills. Similarly, in Sequential Editing for Lifelong Training of Speech Recognition Models, Devang Kulshreshtha et al. apply scaling factors to integrate new domain knowledge sequentially without displacing existing weights.
More sophisticated techniques move beyond global scaling to perform localized adjustments. Peng Ye introduces EMR-Merging in EMR-Merging: Tuning-Free High-Performance Model Merging, which uses binary masks to correct sign conflicts and scalar values to adjust the magnitude of unified task vectors. To minimize the discrepancy in internal representations between individual expert models and the merged model, Wenju Sun et al. propose layer-wise optimal task vector merging (LOT Merging) in Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration, ensuring that the internal feature distributions remain stable.
Key Foundational Work
Learn From Model Beyond Fine-Tuning: A Survey (2023) — contributed 2 concept elements to this direction.
Editing models with task arithmetic (2023) — contributed 2 concept elements to this direction.