Route to Train, Average to Deploy: Federated Multi-Task Adapters

When many organizations want to adapt one large pre-trained model to different tasks without sharing their data, they face a choice. They can average everyone's updates into one model, which is cheap to run but lets conflicting tasks pull the model in opposite directions. Or they can keep a pool of specialist modules and a learned selector that picks among them for every input, which handles conflicts but makes every prediction more expensive. This note proposes testing a third option. Clients train with a learned selector so that conflicting tasks write to different modules. Once training ends, the selector is discarded and the modules are averaged into a single small add-on whose inference cost is the same as plain fine-tuning.

The bet rests on a centralized finding. Multi-Head Adapter Routing (MHR) trains small low-rank adapter pieces with a learned per-task mixture over them. It reports that the mixing mainly helps during training, by aligning task gradients and reducing negative transfer. It also reports that simply averaging the adapter parameters afterward (its MHR-μ variant) keeps cross-task generalization, improving zero-shot accuracy by up to 3% over T0-11B. Federated work has gone the other way. FedLEASE keeps a router active at inference over clustered experts and argues that this runtime routing is needed. The open question is whether the MHR result survives federation, where clients are task-skewed, see only their own data, and are combined by averaging at a server.

Core Claim

In federated fine-tuning across clients that hold different tasks, a learned mixture over shared adapter modules should be used as an aggregation device, not as an inference device. Clients learn local mixing weights over a shared inventory of adapter "skills." The server averages each skill's updates weighted by how much each client actually used that skill. Deployment then collapses the inventory into one adapter, either a uniform average for new clients and tasks or a client-specific weighted average for known clients. We claim this collapsed model will come within about one point of runtime-routed federated experts such as FedLEASE on GLUE and FLAN. We also claim it will beat FedAvg over a single adapter with the same parameter budget. Finally, we claim the mechanism is measurable: usage-weighted aggregation reduces how often clients' updates to the same parameter disagree in sign, which is the interference that TIES-Merging identified in centralized merging.

Relation to Prior Work

Some prior work motivates the proposal. MHR supplies the method: splitting LoRA (low-rank adaptation, a frozen backbone plus small trainable low-rank weight matrices) into heads with learned routing, then averaging. Its regime is centralized only, with full access to the T0 and SuperNI corpora. Two federated works keep the specialists at test time. FedLEASE clusters clients by their LoRA matrices after a warmup and routes among the experts per input. FedTreeLoRA builds a fixed tree in which shallow layers are shared and deep layers branch, and each client blends a cluster expert with an external expert. In both, the structure is fixed after warmup. We learn the client-to-skill assignment continuously, and we test whether any specialist structure is needed at inference at all.

Other work constrains the design. TIES-Merging and Representation Surgery both show that naively merging task models fails. TIES traces the failure to sign conflicts and redundant parameters. Representation Surgery traces it to shifted internal features: Task Arithmetic on ViT-B/32 reaches 69.1% against 88.9% for joint training. This is the strongest evidence against our collapse step. We answer it by having clients co-train against a shared inventory during training, rather than merging independently trained models after the fact.

A third group of federated work leaves the question untouched. FedRAM reweights tasks and clients through a proxy model, but trains full-parameter models and yields one global model. Fed-ensemble and FedBE ensemble whole models in output or posterior space, from scratch and on single tasks. FedTune shows that parameter-efficient federated fine-tuning is cheap and regularizing, reporting 17.3 KB per round with bias tuning, but it studies single tasks only.

Research Program

The backbone stays frozen. A shared inventory of $m$ LoRA skills is attached to each adapted layer, and each skill is split into $h$ heads as in MHR. Each client $c$ keeps a private mixing matrix $z_c$, where $z_{c,j}^{(k)} \ge 0$ is the weight head $k$ places on skill $j$, with weights summing to one over $j$. In each round, the client trains both its mixing matrix and the skill parameters on its local tasks. It then uploads two things: its skill deltas $\Delta_{c,j}$ and the $m \times h$ usage totals $u_{c,j} = n_c \sum_k z_{c,j}^{(k)}$, where $n_c$ is the client's sample count. The mixing matrix itself never leaves the client. The server updates each skill with a usage-weighted average:

$$\Delta_j = \frac{\sum_c u_{c,j}\,\Delta_{c,j}}{\sum_c u_{c,j}}.$$

Under this rule, a client that barely uses skill $j$ barely moves it. Two clients with conflicting tasks can therefore settle on different skills rather than cancelling each other out on one shared adapter.

At deployment, the inventory collapses into one adapter. For an unseen task or a new client, each skill's parameters are averaged with weights $\bar u_j$, which is MHR-μ with weights given by global usage. For a known client, the client's own mixing weights produce a personal adapter. In both cases the result can be folded into the backbone weights, so inference costs exactly what single-LoRA inference costs.

Alongside accuracy, we will log two diagnostics each round. The first is the fraction of coordinates on which client deltas to the same skill disagree in sign. The second is how concentrated client usage is across skills. Together they let us check whether accuracy gains coincide with lower conflict, as the mechanism predicts.

Baselines

The naive reference points are two. The first is isolated local LoRA fine-tuning with no communication. The second is FedAvg over a single LoRA adapter, in the style of FedIT, with rank set so its trainable parameter count matches our full inventory. FedTreeLoRA reports beating this FedIT-style baseline by 5.16% on GLUE with RoBERTa-Large. That paper's absolute FedIT number is not in the file, so we will rerun it ourselves.

The strongest existing competitor is FedLEASE. It is the method that most directly claims runtime routing over federated LoRA experts is necessary. It reports 87.76% average accuracy on GLUE with RoBERTa-Large and 61.70% average ROUGE-1 on FLAN with LLaMA-2-7B. We will run it on identical client partitions, with the same backbone, number of rounds, local steps, and total trainable parameters. Its $M$ experts will be matched to our $m$ skills. Every method gets the same hyperparameter sweep budget. FedTreeLoRA is a second structured competitor and will be run the same way.

Beating FedLEASE does not require exceeding its accuracy. The bar is accuracy within one point of it, at strictly lower inference compute and with no router. Exceeding its accuracy on held-out tasks would be a clear win.

The key ablations isolate routing. The first freezes $z_c$ at uniform values, which keeps the skills but removes specialization. The second fixes $z_c$ at random values. The third keeps routing but aggregates every skill with plain sample-count weights instead of usage weights. A final ablation compares the collapsed model with the same model run with its routing kept at inference. That comparison directly measures what discarding the router costs.

Why not just…?

Why not just run FedAvg on one larger LoRA? FedAvg is the cheapest option, and if it matches us the proposal fails. The evidence suggests it will not. FedRAM reports that uniform aggregation lets large tasks dominate, losing at least 3% F1 on multi-task suites. FedTreeLoRA reports a 5.16% loss against structured LoRA on GLUE. Neither result is parameter-matched to our setup, so the parameter-matched FedAvg baseline is what settles this question.

Why not just train locally once and TIES-merge at the server? One-shot merging saves every communication round. However, Representation Surgery shows that post-hoc merging leaves a large gap to joint training, 69.1% against 88.9% for Task Arithmetic. TIES-Merging itself still trails joint multi-task training. We include one-shot TIES as a baseline. A real threat is a hybrid that applies TIES trimming and sign election to single-adapter aggregation in every round. We run that hybrid too. If it closes the gap, routing adds nothing beyond conflict resolution, and that is the conclusion we will report.

Why not just keep FedLEASE's router at test time? If collapsing costs several points, the router is worth its price, and MHR's centralized finding does not carry over to federated training. The ablation that compares routed and collapsed versions of our own model measures this directly, holding everything else fixed. We will report inference FLOPs and latency alongside accuracy so that the trade-off is explicit.

Proposed Experiments

We will use three benchmarks. The first is GLUE (MNLI, QNLI, SST-2, QQP) with RoBERTa-Large, with clients partitioned by task as in FedLEASE and FedTreeLoRA. The second is FLAN with LLaMA-2-7B, scored by ROUGE-1. The third is the FedRAM and TIES multi-task NLP suite, including tasks such as RTE, CB, COPA, WiC, and Winogrande. For that suite, several tasks are held out entirely to test zero-shot transfer of the uniformly collapsed adapter. TIES reports 4.4% out-of-domain gains with T5-large on this suite, which serves as a reference point. We will additionally vary task overlap across clients, from one task per client to mixtures, and the number of skills $m$.

The decisive checks are three. First, on GLUE and FLAN, the collapsed model must land within one point of FedLEASE (87.76% accuracy and 61.70 ROUGE-1 as reported) with no routing at inference, and must beat parameter-matched FedAvg-LoRA. Second, the uniform-routing ablation must lose most of that gain, and the per-skill sign-conflict rate must be measurably lower under usage-weighted aggregation. Without both, the mechanism is not what we claim. Third, on held-out tasks, the uniformly collapsed adapter must match or exceed the routed variant, which mirrors the MHR-μ result in the federated setting.

If the per-round TIES hybrid matches us on all three checks, the right conclusion is that federated multi-task interference is a sign-conflict problem rather than a routing problem. That would be worth publishing as the negative result.