Task Arithmetic in Orthogonal Parameter Subspaces

This direction is merged into another direction: parameter interference. Its last version stays readable and citable here.

Version 1 of 169 papers95 conceptsfrozen 2026-10-04

When we fine-tune a large language model on a new task, we might assume we are fundamentally reshaping its millions of parameters. However, recent research suggests that fine-tuning actually occurs within "tiny," highly specialized subspaces of the model’s total parameter space. Because these subspaces are often nearly orthogonal to one another, we can treat the knowledge gained during training as a vector—a "task vector"—that can be added, subtracted, or merged with others without causing the model to collapse.

The Geometry of Task Vectors

The core intuition behind this research direction is that the "knowledge" a model acquires during fine-tuning can be represented as a displacement vector in weight space: $\tau = \theta_{fine-tuned} - \theta_{pre-trained}$. Gabriel Ilharco and colleagues demonstrated in Editing Models with Task Arithmetic that these vectors can be manipulated like simple numbers. You can add a "translation" task vector to a base model to give it new capabilities, or subtract a "toxicity" vector to reduce undesirable behaviors.

This works because of weight disentanglement. As Zhong Zhang and Zhenfei Yin show in Fine-tuning Happens in Tiny Subspaces, different tasks naturally occupy nearly orthogonal directions. In the high-dimensional landscape of a neural network, there is enough "room" for a model to learn a new skill in a direction that doesn't interfere with its existing knowledge. Adam Hazimeh and Alessandro Favero further proved that this disentanglement isn't just a quirk of specific models but a general property that emerges as models are pre-trained at sufficient scale.

Why Orthogonality Prevents Interference

If two task vectors are orthogonal, their dot product is zero. Geometrically, this means that moving the model's weights in the direction of Task A has zero projection onto the direction of Task B. Feng Xiong and Chun Yuan provide a mathematical foundation for this in Multi-Task Model Merging via Adaptive Weight Disentanglement, showing that the performance degradation during model merging is minimized precisely when these task vectors are orthogonal.

This structural independence is what allows for "zero-cost" model merging. Ilja Kuzborskij and Yasin Abbasi-Yadkori explain that standard training techniques like weight decay induce a "low-rank bias," forcing models into narrow, low-dimensional subspaces. When models are trained on different tasks with nearly orthogonal inputs, their weight structures do not overlap, allowing them to be merged via simple summation.

Engineering Better Subspaces

While orthogonality often happens naturally, researchers have developed methods to explicitly enforce or enhance it to allow for more complex model compositions:

  • Linearization: Anke Tang and Li Shen introduced L-LoRA, which applies partial linearization to adapters. This increases the geometric orthogonality of task vectors, making task arithmetic more precise. Similarly, Guillermo Ortiz-Jiménez found that fine-tuning in the "tangent space" of a model improves the effectiveness of weight edits.
  • Sparsity and Non-collision: Kun-Peng Ning and Yi-Bing Song argue in Is Parameter Collision Hindering Continual Learning in LLMs? that the ultimate form of orthogonality is parameter non-collision. By using sparsity constraints (N-LoRA) to ensure different tasks use entirely different physical "slots" in the weight matrix, they achieve a 58x reduction in interference.
  • Random Projections: Methods like FlyLoRA by Heming Zou and LoRI by Juzheng Zhang use fixed, random projection matrices for different tasks. Since random vectors in high dimensions are almost always nearly orthogonal, this ensures that task updates are statistically unlikely to overlap.

Applications Across Domains

The ability to treat model weights as composable blocks has led to breakthroughs in several areas:

  • Continual Learning: Hao Zhu and colleagues developed BiLoRA, which uses orthogonal bases to expand the available parameter space, allowing models to learn a sequence of tasks without "catastrophic forgetting."
  • Multi-Modal Fusion: Lingjie Jiang and Shaohan Huang demonstrated in VisCodex that vision-language processing and programming logic occupy disjoint subspaces, allowing a vision model and a coding model to be arithmetically merged into a single multimodal programmer.
  • Safety and Watermarking: Siyu Yuan showed that watermark signatures and task-specific knowledge reside in orthogonal directions, meaning you can embed a statistical "fingerprint" into a model's weights without degrading its reasoning capabilities.

By shifting the view of model training from "global weight updates" to "navigation within task-specific subspaces," this research direction provides a scalable, modular way to build and edit the next generation of foundation models.

Papers in this version

Showing 60 of 69 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_task_arithmetic_in_orthogonal_parameter__30471003,
  title = {Task Arithmetic in Orthogonal Parameter Subspaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-arithmetic-in-orthogonal-parameter-subspaces/rdr_30471003d5674a039784d0ccb84616c6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_task_arithmetic_in_orthogonal_parameter__30471003_v1,
  title = {Task Arithmetic in Orthogonal Parameter Subspaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-arithmetic-in-orthogonal-parameter-subspaces/rdr_30471003d5674a039784d0ccb84616c6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}