Functional Asymmetry in Low-Rank Adaptation

54 concepts36 papers1 proposal

When fine-tuning large language models, Low-Rank Adaptation (LoRA) has become the default strategy for parameter efficiency. In its standard formulation, LoRA decomposes a weight update $\Delta W$ into the product of two low-rank matrices, $B \cdot A$, where $A$ compresses the input dimension to a small rank $r$, and $B$ expands it back to the model's hidden dimension. Traditionally, these two matrices are treated symmetrically: they are optimized together, scaled together, and allocated equally across tasks.

However, a deeper look into the geometry of low-rank updates reveals a profound functional asymmetry. Matrices $A$ and $B$ are not doing the same job. Matrix $A$ acts as an encoder or feature extractor, mapping high-dimensional representations into a low-rank bottleneck. Matrix $B$ acts as a decoder or task mapper, translating those compressed features into specific behavioral changes. Recognizing and exploiting this asymmetry allows for a new class of highly efficient, modular, and robust adaptation architectures.

Why the Input Matrix is a Generalist

To understand why this asymmetry exists, we must look at the intrinsic dimensionality of foundation models. Pre-trained models already possess rich, highly structured representations of data. When adapting to a new task, the model does not need to learn entirely new feature extractors from scratch; it primarily needs to select and recombine existing features.

This leads to a surprising geometric property: the down-projection matrix $A$ is largely a generalist. As demonstrated by Jiacheng Zhu et al., if you freeze a completely random, untrained $A$ matrix and only fine-tune $B$ (with a slightly higher rank), the model can match or even exceed standard LoRA performance. Because the model's intrinsic feature space is low-dimensional, a random projection is sufficient to capture the necessary signals.

Empirical evidence from Lin Ning 0001 et al. confirms that when training standard LoRA adapters across highly diverse tasks, the learned down-projection matrices ($A$) remain remarkably consistent and redundant across domains. The actual task-specific specialization happens almost entirely within the up-projection matrix ($B$), which undergoes significantly larger updates during training to map those compressed features to the target output space. Similarly, Nadav Benedek showed that structural pruning can be focused heavily on the input-side matrix ($A$) because it acts as the primary compression bottleneck, leaving the output-side matrix intact to preserve expressive capacity.

Breaking Symmetry for Multi-Task Experts

If the input matrix $A$ learns general feature representations while the output matrix $B$ handles task-specific specialization, duplicating both matrices for every new task or expert is highly redundant. Instead, we can break the symmetry by sharing a single feature-extracting bottleneck across multiple specialized heads.

Standard LoRA (Symmetric)        Asymmetric Shared-Feature Bottleneck
     [ Input X ]                             [ Input X ]
      /       \                                   |
  [Task 1 A] [Task 2 A]                       [Shared A]
      |          |                                /      \
  [Task 1 B] [Task 2 B]                       [Expert B1] [Expert B2]
      \          /                                \      /
     [ Output Y ]                            [ Output Y ]

This architectural shift enables highly compact Mixture-of-Experts (MoE) and multi-task frameworks:

  • Shared-Input Experts: Frameworks like HydraLoRA, introduced by Chunlin Tian et al., and BranchLoRA, developed by Duzhen Zhang et al., utilize a single shared input projection matrix to capture task-invariant patterns, while employing multiple task-specific output matrices to handle domain nuances.
  • Hierarchical MoE: Mengqi Liao et al. (HMoRA) and Haoyuan Li et al. (TeamLoRA) apply this to MoE routing, sharing the input projection across all experts to drastically reduce the parameter footprint while maintaining distinct specialized behaviors.
  • Noise Isolation: Zhaokun Wang et al. exploit this structure to achieve noise-robustness, using a shared input matrix with independent output experts to isolate and mask data noise during inference.

Conversely, the sharing strategy can be inverted depending on the optimization goal. For instance, Hao Ban et al. argue that in certain multi-adapter configurations, sharing the $B$ matrix as a centralized knowledge repository while using task-specific or routed $A$ matrices can minimize gradient conflicts, since $B$ acts as the primary repository of task-specific knowledge.

Cross-Layer and Personalized Bottlenecking

The economy of sharing the generalist matrix extends beyond multi-tasking to cross-layer optimization and personalization.

In EffiLoRA, Chunlin Tian et al. take parameter reduction to its logical extreme by sharing a single down-projection matrix across all transformer layers, while keeping the up-projection matrices distinct. This exploits the fact that down-projections across different layers often converge to similar subspaces, allowing a universal semantic bottleneck to serve the entire network.

For personalized reward modeling in RLHF, Renpu Liu et al. demonstrate that sharing the up-projection component across multiple users captures common structural preference directions, while individual, private down-projection matrices efficiently capture user-specific tastes.

Asymmetric Dynamics in Continual and Distributed Learning

The functional split between $A$ and $B$ also provides a natural solution to the stability-plasticity dilemma in continual learning and the aggregation challenge in federated learning.

In continual learning, Fuli Qiao et al. show that because the matrices follow different learning dynamics, one can update the input matrix to the latest task's state while sequentially merging the output matrix. This preserves geometric consistency across tasks without catastrophic forgetting. Similarly, Omayma Moussadek et al. decouple the updates to freeze task-specific subspaces, balancing stability and plasticity.

In federated learning, decentralized clients often suffer from data heterogeneity. Frameworks like UniFLoW (Haoyuan Liang et al.) and strategies by Pengxin Guo et al. exploit structural asymmetry by treating the input projection matrix as a stable global anchor for feature extraction, while allowing the output projection matrix to handle local adaptation. This enables more sophisticated, targeted aggregation strategies that improve global convergence while preserving local specialization. Furthermore, Han Liu et al. observe that the matrix initialized to zero exhibits asymmetric learning dynamics by becoming significantly sparser, allowing for highly targeted communication compression.

Key Foundational Work

Asymmetry in Low-Rank Adapters of Foundation Models (2024) — contributed 3 concept elements to this direction.
LoRA+: Efficient Low Rank Adaptation of Large Models (2024) — contributed 3 concept elements to this direction.
HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_low_rank_adaptation_0cada13d,
  title = {asymmetric low-rank adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-low-rank-adaptation/rdr_0cada13daa684c37b5ac3063b42affc9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_asymmetric_low_rank_adaptation_0cada13d_v1,
  title = {asymmetric low-rank adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-low-rank-adaptation/rdr_0cada13daa684c37b5ac3063b42affc9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake