Low-Rank Adaptation (LoRA) has become the standard for fine-tuning foundation models by decomposing weight updates into two small matrices, $A$ and $B$, such that $\Delta W = BA$. While the original formulation treated these matrices as symmetric components of a product, a growing body of research reveals a fundamental functional asymmetry: the two matrices play distinct roles in how a model learns new information.
By understanding and exploiting this asymmetry, researchers are building models that are more efficient, better at multi-tasking, and easier to personalize in decentralized environments.
The Encoder-Decoder Intuition
The core of the asymmetry lies in the mathematical "direction" of the matrices. In a standard LoRA layer, the $A$ matrix ($d \times r$) acts as a down-projection or encoder, while the $B$ matrix ($r \times d$) acts as an up-projection or decoder.
As Asymmetry in Low-Rank Adapters of Foundation Models demonstrates, the $A$ matrix serves as a feature extractor that identifies the relevant subspace of the input data. Conversely, the $B$ matrix maps these extracted features to the target output. This research found that because foundation models have low intrinsic dimensionality, even a random, frozen $A$ matrix can often capture enough feature information for a trained $B$ matrix to achieve high performance. This suggests that $B$ is the "heavy lifter" for task-specific adaptation, while $A$ captures more general structural information.
Eliminating Redundancy in Multi-Task Learning
In multi-task scenarios, standard LoRA often suffers from the "seesaw effect," where learning one task degrades performance on another. Researchers have found that this is often due to parameter redundancy: multiple task-specific adapters often learn nearly identical feature extraction patterns in their $A$ matrices.
To solve this, Chunlin Tian and others proposed architectures like HydraLoRA and AsymLoRA. These models use a single, shared $A$ matrix to capture domain-agnostic features across all tasks, while employing multiple specialized $B$ "heads" to handle task-specific nuances.
This "one-to-many" structure—one $A$ for many $B$s—offers several advantages:
- Parameter Efficiency: MALoRA shows that sharing the down-projection can reduce parameter counts by up to 48% without losing accuracy.
- Reduced Interference: By forcing the model to learn shared representations in $A$, the specialized $B$ matrices can focus purely on task-specific logic, reducing conflicts between tasks.
- Inference Speed: TeamLoRA leverages this asymmetry to speed up inference, as the model only needs to perform the $A$ projection once before routing to different experts.
Personalization via Selective Aggregation
The functional split between $A$ and $B$ is particularly powerful in Federated Learning (FL), where models are trained on private, decentralized data. In these settings, there is a constant tension between learning "global" knowledge that applies to everyone and "local" knowledge that fits a specific user.
Pengxin Guo and Shuang Zeng have shown that the $A$ matrix typically captures the stable, global features of a task, while the $B$ matrix adapts to the specific data distribution of a local client. In Selective Aggregation for Low-Rank Adaptation in Federated Learning, they propose sharing only the $A$ matrix globally while keeping $B$ local.
Similarly, FedPissa by Wenwen He uses this asymmetry to balance global collaboration with local personalization. By treating the matrices as functionally distinct, these frameworks achieve better convergence and higher accuracy than methods that try to aggregate the entire adapter.
Stability in Continual Learning
When a model learns tasks sequentially, it often "forgets" earlier tasks—a problem known as catastrophic forgetting. Functional asymmetry provides a mechanism to anchor the model's geometry.
In Merge before Forget: A Single LoRA Continual Learning via Continual Merging, Fuli Qiao and Mehrdad Mahdavi introduce SLAO. This method treats the $A$ and $B$ matrices as asymmetric entities with different learning dynamics. By updating the input-facing $A$ matrix to reflect the latest task while carefully merging the output-facing $B$ matrix to preserve historical knowledge, they enable a single LoRA adapter to learn a sequence of tasks without the typical linear growth in parameters.
Why It Works: The Sensitivity Gap
The reason these asymmetric strategies are so effective is that the two matrices exhibit different sensitivities to optimization. FedLoRA-Optimizer reveals that the $A$ matrix primarily captures the direction of the weight update, while the $B$ matrix dominates the magnitude.
This sensitivity gap means that the $A$ matrix is more critical for defining the "what" (the feature subspace), while $B$ is more critical for the "how much" (the intensity of the feature's application). As Taha Ceritli notes in A Study of Parameter Efficient Fine-tuning, prioritizing the configuration of the input projection ($A$) leads to more significant improvements in task accuracy than focusing on the expansion matrix ($B$).
By moving away from the view of LoRA as a symmetric product and embracing the distinct roles of $A$ and $B$, the field is shifting toward "surgical" fine-tuning—where parameters are shared, localized, or frozen based on the specific functional role they play in the model's adaptation.