Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning

Yebo Wu, Jingguang Li, Zhijiang GuoSubmission to ICLR 2026 · Withdrawn

TL;DR

Federated fine-tuning of Large Language Models (LLMs) on edge devices is often crippled by high computational and communication costs. SMARTFED eliminates training from scratch by decomposing pre-existing, task-specific LoRA modules into rank-one "experts" and training only a lightweight gating router across devices. Paired with an elastic quota allocation strategy that redistributes expert capacity where it matters most, SMARTFED slashes communication by up to $31.47\times$ and accelerates convergence by $3.95\times$ while outperforming traditional federated fine-tuning baselines.

What Problem Does This Paper Address?

Adapting LLMs to private downstream data across distributed devices (like mobile phones or hospital servers) requires federated learning. However, even parameter-efficient fine-tuning (PEFT) methods like LoRA are prohibitively expensive when trained from scratch on resource-constrained devices, requiring hundreds of rounds of gradient communication.

While thousands of specialized LoRA adapters are openly available in model hubs, reusing them across tasks is surprisingly difficult:

  1. Linear parameter merging (e.g., adding adapters together) produces noisy cross-task interference terms that degrade performance.
  2. Mixture of LoRA Experts (MoLE) treats entire low-rank matrices as coarse-grained experts, introducing significant inference latency and failing to isolate specific, fine-grained representations.

SMARTFED tackles the challenge of efficiently reusing existing LoRA knowledge in privacy-preserving federated settings without the prohibitive overhead of full retraining.

Methods

SMARTFED introduces two complementary techniques that transform federated adaptation into a sparse, rank-level routing problem:

[Public LoRA Repository]
          │
          ▼  (Decompose rank r into r rank-1 vectors)
[Rank-Wise Expert Pool: E_i(x) = B_{:,i} * A_{i,:} * x]
          │
          ▼  (Train only router locally on edge devices)
[Top-K Sparse Dynamic Gating] ──► [Elastic Expert Quota Allocation (EEQA)]
  • Mixture of Rank-Wise Experts (MoRE): Instead of treating a rank-$r$ LoRA update ($\Delta W = BA$) as a single monolithic block, MoRE decomposes it along its rank dimension into $r$ independent, rank-one projections: $$E_i(x) = (B_{:,i} A_{i,:}) x$$ The server freezes these decomposed experts and distributes them. Edge clients train only a lightweight token-level router ($W_{\text{router}}$), which dynamically selects and weights the top-$K$ experts based on input semantics.

  • Elastic Expert Quota Allocation (EEQA): Different transformer matrices (e.g., Query vs. Value) and layers do not contribute equally to task performance. EEQA tracks client-side cosine-similarity deviations between input embeddings and expert outputs to calculate an importance score. The server dynamically allocates the number of active expert slots ($q_j$) per matrix under a fixed total compute budget, prioritizing crucial layers over less sensitive ones.

Datasets

The authors evaluate skill composition across three combinations:

  • Chinese Mathematical Reasoning (Chinese + Math): Trained on Math23K (partitioned across 20 simulated devices) and tested on MGSM using accuracy.
  • Chinese Code Generation (Chinese + Code): Evaluated on the DoIT benchmark (Pass@1 metric), with a training set expanded to 20,000 samples via web crawling and GPT-4 verification before partitioning.
  • Hard Math–Word Problems (Math + Code): Trained on MathCodeInstruct and evaluated on GSM-Hard via execution accuracy.

Underlying skill-specific LoRAs were pre-trained on the Okapi Chinese instruction dataset, MetaMath (395K English math problems), and Magicoder (186K coding problems).

Key Insights and Ideas

  • Rank-1 Subspaces as Modular Knowledge Units: LoRA matrices pack multiple latent directions into a single adapter. Splitting them into rank-1 components eliminates destructive cross-term interference ($\lambda_1 \lambda_2 B_1 A_2 x$) while enabling the model to select specialized sub-capabilities independently.
  • Non-Uniform Matrix Sensitivity: Equal quota allocation across all attention matrices is suboptimal. Empirical analysis showed that Query matrices often demand much higher rank allocations than Value matrices, and importance varies wildly across layers (e.g., allocating 58 experts to Layer 16 Value, but only 1 to the corresponding Query matrix).
  • Extreme Data Efficiency: Because the core domain capabilities are already embedded within the pre-trained experts, clients only need to learn how to route. On LLaMA2-7B, SMARTFED using only 10% of local training data outperformed standard FedIT trained on 100% of the data by up to 8.02%.

Key Results

  • Superior Downstream Performance: Tested across LLaMA2-7B, LLaMA2-13B, and Qwen2-7B, SMARTFED consistently outperformed both knowledge-free federated baselines (FedIT, FwdLLM, DoFIT, AdaLoRA) and centralized adapter-merging methods (LoRAHub, MoLE, LoRA-Flow), achieving up to a 10.21% average gain on LLaMA2-13B.
  • Massive Resource Savings: On LLaMA2-7B (Chinese + Math):
  • $3.95\times$ faster training time (0.95 hours vs. 3.75 hours for FedAdapter).
  • $31.47\times$ lower communication footprint (0.81 GB total transmitted vs. 25.49 GB for FwdLLM and 4.99 GB for FedIT).
  • $3.61\times$ reduced energy consumption (1.52 kWh vs. 5.49 kWh for FwdLLM).
  • Ablations: Removing MoRE led to a 4.23%–4.95% average drop, while removing EEQA caused a 3.31%–3.45% drop, confirming that both fine-grained rank decomposition and non-uniform allocation are critical.

Limitations and Future Work

  • Pre-trained LoRA Availability: SMARTFED relies on the availability of pre-existing, high-quality LoRA modules matching the constituent skills of the target task.
  • Skill Scaling: Experiments tested compositions of two primary skills at a time; routing dynamics when scaling to dozens of retrieved LoRA modules simultaneously remain unaddressed.
  • Communication of Expert Quotas: While router communication is minimal, the protocol requires round-trip transmissions of scalar importance scores for quota rebalancing.

Summary updated

Abstract

Federated fine-tuning offers a promising solution for adapting Large Language Models (LLMs) to downstream tasks while safeguarding data privacy. However, its high computational and communication demands hinder its deployment on resource-constrained devices. In this paper, we propose SmartFed, a resource-efficient federated fine-tuning framework. SmartFed intelligently reuses knowledge embedded in existing LoRA modules, eliminating the need for expensive training from scratch when adapting LLMs to new tasks. To effectively exploit this knowledge and ensure scalability, we introduce the Mixture of Rank-Wise Experts (MoRE). MoRE decomposes LoRA modules into fine-grained rank-level experts. These experts are selectively activated and combined based on input semantics and resource budgets. Moreover, to optimize resource utilization, we present the Elastic Expert Quota Allocation (EEQA). EEQA adaptively allocates expert capacity across parameter matrices based on their contribution to model performance, focusing computing resources on the critical experts. Extensive evaluations across multiple benchmarks demonstrate that SmartFed significantly outperforms existing methods in model performance and training efficiency.
BibTeX
@misc{wu2025elastic,
  author = {Yebo Wu and Jingguang Li and Zhijiang Guo and Li Li},
  title = {{Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning}},
  year = {2025},
  eprint = {2512.00902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.00902},
}
Figure