Multi-Head Adapter Routing for Cross-Task Generalization

Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, Alessandro Sordoni

TL;DR

The paper introduces Multi-Head Routing (MHR), a parameter-efficient fine-tuning method that achieves state-of-the-art cross-task generalization by routing sub-blocks of adapter parameters (heads) rather than whole modules. Crucially, the authors demystify why routing works, demonstrating that its primary value lies in improving gradient alignment and reducing task interference during multi-task pre-training, rather than facilitating modular recombination at test time. This insight allows them to discard routing during downstream adaptation entirely, simplifying deployment into a single averaged adapter (MHR-$\mu$) or tuning only the routing weights (MHR-$z$) for extreme parameter efficiency.

What Problem Does This Paper Address?

When adapting large pre-trained language models to new tasks with very few examples (few-shot learning), full fine-tuning is computationally prohibitive. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA solve this by training small "adapter" modules while keeping the backbone model frozen.

To generalize well to unseen tasks, researchers often pre-train these adapters on a diverse mix of training tasks (multi-task pre-training) before few-shot adaptation. However, this strategy frequently suffers from negative transfer—where different tasks have conflicting gradients, causing them to interfere with and degrade each other's performance.

Prior work, such as Polytropon (Poly), attempted to solve this by learning an inventory of adapter modules and a routing function to select a subset of modules per task. However, Poly operates at a coarse level (routing entire adapters) and relies on the unverified assumption that its success stems from the test-time recombination of these modules. This paper addresses two core questions:

  1. Can we make adapter routing more expressive without exploding the parameter budget?
  2. What is the actual underlying mechanism driving the success of routing-based PEFT?

Methods

The authors propose Multi-Head Routing (MHR), which applies the intuition of multi-head attention to adapter routing.

Instead of making a single routing decision for an entire LoRA adapter, MHR partitions the adapter parameter matrices ($A$ and $B$) into $h$ disjoint blocks or "heads." For each head, a separate routing function selects and linearly combines modules from an inventory of $m$ available copies. These task-specific heads are then concatenated together to form the final LoRA adapter layer.

Figure 1: Left: A LoRA adapter with weight AB⊤is trained on top of a frozen, pre-trained linear layer W. Our method MHR partitions the A, B parameter indexes into h blocks (or heads). For each block, a separate routing function selects the active modules for the current task among m copies with different parameter values, and combines them via averaging to form a task-specific head. The heads are then concatenated to form the LoRA adapter. Using multiple heads allows for more fine-grained mixing of task parameters with a negligible increase in overall parameter count. Right: During few-shot adaptation, one can fine-tune only the multi-head routing parameters (MHR-z), keeping the modules frozen, resulting in highly parameter-efficient adaptation.

Building on this architecture, the authors introduce two key variants:

  • MHR-$z$ (Routing-Only Fine-Tuning): During downstream few-shot adaptation, the adapter modules are completely frozen. The model only updates the low-dimensional routing weights ($Z$), making it incredibly parameter-efficient.
  • MHR-$\mu$ (Adapter Average Fine-Tuning): The routing function is discarded at test time. Instead, the parameters of all pre-trained modules are uniformly averaged into a single standard LoRA adapter before few-shot fine-tuning.

Datasets

The methods were evaluated on two major cross-task generalization benchmarks using T5-XL (3B) and T0 (3B and 11B) backbones:

  1. T0 Task Suite: A collection of prompt-augmented datasets spanning multiple NLP tasks (e.g., sentiment analysis, question answering). The setup treats each dataset-template pair as a unique task (313 tasks total).
  2. Super-Natural Instructions (SuperNI): A large-scale benchmark containing over 1,600 tasks. The authors evaluated generalization on a randomized subset of 20 downstream test tasks.

Key Insights and Ideas

  • Granular Routing Scales Better Than More Modules: Increasing the number of routing heads provides a piecewise-linear combination of parameters. This drastically increases model expressivity and performance at a fraction of the parameter cost compared to simply adding more standalone adapter modules.
  • Routing is an Optimization Helper, Not a Modularity Provider: The most surprising "aha!" moment of the paper is that routing's main benefit occurs during multi-task pre-training, not during downstream adaptation. MHR acts as an optimization mechanism that maximizes gradient cosine similarity between different training tasks, effectively mitigating negative transfer.
  • Test-Time Routing is Optional: Because routing primarily fixes pre-training optimization, you don't actually need to route modules at test time. Averaging the modules (MHR-$\mu$) performs just as well as dynamic routing, meaning practitioners can deploy a single, standard adapter downstream without any routing overhead.

Key Results

MHR consistently outperforms standard LoRA and Polytropon across benchmarks:

  • T0 Generalization (T5-XL backbone): MHR achieves 69.1% average accuracy, outperforming Polytropon (68.0%) and standard LoRA (66.0%).
  • Extreme Parameter Efficiency (MHR-$z$): When freezing the adapters and tuning only the routing parameters during few-shot adaptation, MHR-$z$ achieves 68.3% accuracy on T0. It beats the standard Poly baseline while using orders of magnitude fewer adaptation parameters (220K vs 2.2M).
Figure 2: Left: Results of few-shot adaptation on T0 dataset Sanh et al. [2022]. We report the mean of the best validation accuracy for each test task. Subscripts correspond to standard deviation. Right: Accuracy of PEFT methods on the T0 dataset when applied on top of T5-XL. The x-axis shows the parameter count during the fine-tuning process.
  • Zero-Shot Excellence: By taking the averaged adapter from MHR-$\mu$ and training it for a few additional steps on the multi-task training set, it achieves a zero-shot performance of 64.5% on the T0-11B backbone, outperforming baseline LoRA by up to 3% absolute accuracy.

Limitations and Future Work

While MHR shows impressive gains, the authors note a few areas for improvement:

  • Hyperparameter Tuning: Introducing heads adds another hyperparameter ($h$) to tune, and finding the optimal trade-off between the number of heads and modules requires empirical search.
  • Scope of Architecture: The evaluation focuses primarily on encoder-decoder text-to-text language models. Extending this multi-head routing concept to decoder-only causal LLMs remains an open direction.
  • Broader Applications: The authors suggest that the insights regarding gradient alignment during multi-task training could be applied to improve other modular architectures, such as token-level Mixture of Experts (MoEs).

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) for cross-task generalization consists in pre-training adapters on a multi-task training set before few-shot adaptation to test tasks. Polytropon [Ponti et al., 2023] ($\texttt{Poly}$) jointly learns an inventory of adapters and a routing function that selects a (variable-size) subset of adapters for each task during both pre-training and few-shot adaptation. In this paper, we investigate the role that adapter routing plays in its success and design new variants based on our findings. First, we build on the intuition that finer-grained routing provides more expressivity. Hence, we propose $\texttt{MHR}$ (Multi-Head Routing) which combines subsets of adapter parameters and outperforms $\texttt{Poly}$ under a comparable parameter budget; by only fine-tuning the routing function and not the adapters ($\texttt{MHR}$-$z$) we achieve competitive performance with extreme parameter efficiency. Second, we find that $\texttt{Poly}$/$\texttt{MHR}$ performance is a result of better multi-task optimization, rather than modular inductive biases that facilitate adapter recombination and local adaptation, as previously hypothesized. In fact, we find that $\texttt{MHR}$ exhibits high gradient alignment between training tasks. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation and propose $\texttt{MHR}$-$μ$, which discards routing and fine-tunes the average of the pre-trained adapters on each downstream tasks. This establishes $\texttt{MHR}$-$μ$ as an effective method for single-adapter fine-tuning. We also show that $\texttt{MHR}$-$μ$ can be used as an effective zero-shot transfer method by training the average of the pre-trained adapters for a few additional steps on the multi-task training set: this yields gains up to 3% on absolute accuracy w.r.t. the baselines.
BibTeX
@inproceedings{caccia2023multi,
  author = {Lucas Caccia and Edoardo Ponti and Zhan Su and Matheus Pereira and Nicolas Le Roux and Alessandro Sordoni},
  title = {{Multi-Head Adapter Routing for Cross-Task Generalization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Vn4XRQqjgn},
}
Figure