Rank-Wise Mixture of Experts for Fine-Grained Adaptation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 216 papers21 conceptsfrozen 2026-10-04

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large models by adding a small number of trainable parameters. However, traditional LoRA treats an adapter as a monolithic block: you either use the whole rank-$r$ matrix or you don't.

A new wave of research suggests that this "coarse-grained" approach is inefficient. Instead, researchers are decomposing LoRA adapters into their constituent rank-1 components, treating each individual rank as an independent "expert." By shifting the granularity from the module level to the rank level, models can mix, match, and route knowledge with unprecedented precision.

From Monolithic Blocks to Atomic Ranks

In a standard LoRA layer, the weight update is represented by the product of two matrices, $B$ and $A$. If the rank is $r$, the update is $\Delta W = BA$.

The core insight shared by researchers like Ziyu Zhao and Guanyu Li is that this matrix product is mathematically equivalent to the sum of $r$ independent rank-1 matrices: $$\Delta W = \sum_{i=1}^{r} b_i a_i^T$$ Each pair $(b_i, a_i)$—a single column from $B$ and a single row from $A$—acts as a "Minimal Semantic Unit" or a "memory atom."

In LoRA-LEGO, Ziyu Zhao and colleagues demonstrate that these units are permutation-invariant building blocks. Just as you can build different structures from the same set of LEGO bricks, you can cluster and recombine these rank-1 experts from different models to create a new, specialized adapter without any additional training.

Why Granularity Matters: The Routing Intuition

Why bother breaking a matrix into vectors? The answer lies in interference. When a model learns multiple tasks or encounters diverse image degradations, a single high-rank adapter often tries to capture everything at once, leading to "parameter interference" where knowledge for Task A overwrites Task B.

By treating each rank as an expert in a Mixture-of-Experts (MoE) framework, the model can use a router to activate only the specific ranks needed for a given input.

  • In Image Restoration: Zhijun Tu and Mingrui Zhu developed MoR-DASR, which uses a CLIP-powered router to select ranks based on the specific degradation (like blur or noise) in an image.
  • In Multi-Task Learning: Ziyu Zhao argues in Each Rank Could be an Expert that rank-level MoE allows the model to capture subtle nuances that are lost when using coarse, block-wise experts.

Solving Catastrophic Forgetting

Continual learning—teaching a model new things without it forgetting the old—is a primary beneficiary of this direction. Traditional methods often add a new adapter for every new task, leading to "memory explosion."

Haodong Lu and Lina Yao propose MoRAM, which treats rank-1 components as "associative memory atoms." Instead of an unstable external router, MoRAM uses a self-activation mechanism. As the model learns "little by little," it accumulates these atoms. Because the activation is so fine-grained, the model can retrieve specific knowledge for a new task while leaving the atoms used for old tasks untouched, effectively neutralizing catastrophic forgetting.

Similarly, Weicheng Meng and Yuan Xie introduce PCLR, which progressively compresses these rank-level experts. This allows the model to manage its capacity dynamically, pruning or editing specific ranks to make room for new knowledge.

Efficiency in Federated Learning

The rank-wise approach also solves the bottleneck of communication in federated learning. In SMARTFED, Yebo Wu and Li Li show that instead of training new models on edge devices, we can "realign the spokes" of existing ones.

By decomposing pre-trained LoRAs into rank-wise experts, the framework can adaptively allocate compute resources. A device only needs to download or activate the specific rank-1 experts relevant to its local data. This "expert assembly" approach reduces communication overhead by over 30x because the system is moving small vector pairs rather than large weight matrices.

Bio-Inspired Routing: FlyLoRA

Perhaps the most radical implementation of this idea is FlyLoRA. Heming Zou and Xiangyang Ji mimic the olfactory circuit of a fruit fly to create an "implicit router."

Instead of a complex, trainable routing network, FlyLoRA uses a sparse random projection to activate rank-1 experts. This achieves the benefits of a massive Mixture-of-Experts—high capacity and low interference—without the computational cost usually associated with routing. It proves that when you operate at the granularity of individual ranks, even simple, fixed mechanisms can achieve state-of-the-art task decoupling.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 16 papers
  2. Version 1 · 2026-10-04 · 16 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_rank_wise_expert_granularity_e73b27ce,
  title = {rank-wise expert granularity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rank-wise-expert-granularity/rdr_e73b27ce02b64142a96aa36aa27ba05e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_rank_wise_mixture_of_experts_for_fine_gr_e73b27ce_v1,
  title = {Rank-Wise Mixture of Experts for Fine-Grained Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/rank-wise-mixture-of-experts-for-fine-grained-adaptation/rdr_e73b27ce02b64142a96aa36aa27ba05e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}