Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method

Yimin Tian, Bolin Zhang, Zhiying Tu, Dianhui ChuCOLING 2025 · OpenReview

Abstract

Parameter-Efficient Fine-Tuning (PEFT) adapts large language models (LLMs) to specific domains by updating only a small portion of the parameters. Although fine-tuning on a single task within a specific domain has demonstrated promising results, there remains limited exploration on how to effectively integrate these adapters for optimal performance. In this paper, we propose Adapters Selector (AS): a novel framework for better integrating usage of multiple adapters by training a middleman adapter to select the appropriate adapter for inference. Our approach utilizes PEFT to train a selector that determines which input content corresponds to which task in which domain, and subsequently selects the homologous adapter. The AS has developed the capability to execute cross-domain multi-tasks effectively through the utilization of a compact model in combination with multiple LoRA modules. Our code is publicly available.

Introduction

Adapting Large Language Models (LLMs) to specialized domains often relies on Parameter-Efficient Fine-Tuning (PEFT), specifically Low-Rank Adaptation (LoRA). While effective for individual tasks, a significant challenge arises when a single model must excel across diverse domains like medicine, law, and finance. Simply merging multiple LoRA adapters frequently results in "knowledge conflict," where overlapping weights from different tasks degrade performance. Existing solutions like Multi-task PEFT or Mixture-of-Experts (MoE) LoRA often require complex joint training or suffer from inference delays and data imbalance.

The Adapters Selector (AS) framework addresses this by introducing a "middleman" routing mechanism. Instead of merging weights, AS dynamically identifies the domain and task of an input and switches to the most appropriate expert adapter, preserving the high performance of specialized modules without the overhead of joint training.

Methodology

The AS framework operates through a three-phase pipeline:

  1. Expert Adapter Training: Individual LoRA adapters are trained independently on single tasks within specific domains (e.g., Legal-IE or Medical-MC). This avoids the complexities of multi-task data balancing during the initial training phase.
  2. Selector Training via K-means Selection: A compact LoRA module, the "Selector," is trained to map inputs to specific (Domain, Task) pairs. To ensure a balanced training set, the authors employ K-means clustering on vectors generated by the LLM’s own embedding layer to select representative samples.
  3. Inference Pipeline: During inference, the base LLM first uses the Selector to identify the task. The Selector is then decoupled, and the corresponding expert LoRA adapter is loaded to generate the final response.
Overview of the Adapters Selector framework showing training and inference stages.

Experimental Setup

The framework was evaluated across three primary domains using established benchmarks:

  • Medical (Med): Benchmarks include MedicalNLP (Multiple-Choice) and BioInstruct.
  • Finance (Fin): Evaluated using FinGPT for tasks like sentiment analysis and headline analysis.
  • Legal (Legal): Benchmarks include Legalbench for judgment and information extraction tasks.

The experiments scaled from 6 to 12 tasks to test robustness across various model families, including Qwen2, Gemma, Yi, InternLM, and Llama3.

Results and Discussion

The AS framework consistently outperforms joint-training and MoE-style baselines. In a 6-task configuration, the AS framework (specifically the "Selector 14" configuration) achieved an overall metric of 0.722, significantly higher than MFTCoder (0.607) and MoE LoRA (0.524).

Performance comparison of different selector configurations against baselines in a 6-task setup.

Key insights from the evaluation include:

  • Routing Precision: The Selector's accuracy in identifying the correct task often exceeded 90%, even as task complexity increased to 12 tasks.
  • Self-Embedding Advantage: Using the LLM’s internal embedding layer for data selection outperformed external models like BERT or m3e by 0.5% to 1%, suggesting the model is its own best judge of data similarity.
  • Modular Flexibility: The framework supports "Add," "Balance," and "Delete" operations, allowing for the addition of new tasks without retraining existing experts. For tasks with highly ambiguous boundaries, "deleting" the indistinguishable task from the selector maintained 98.9% performance across the rest of the suite.

Conclusion

While the AS framework provides a robust solution for multi-task integration, it introduces some inference latency due to the two-step switching process (adding a prefill and three-token decoder time). Additionally, tasks with highly ambiguous boundaries within the same domain remain a challenge. Future work will focus on optimizing switching overhead and refining the Selector’s ability to distinguish between closely related tasks.

Summary updated

Abstract

Parameter-Efficient Fine-Tuning (PEFT) adapts large language models (LLMs) to specific domains by updating only a small portion of the parameters. Although fine-tuning on a single task within a specific domain has demonstrated promising results, there remains limited exploration on how to effectively integrate these adapters for optimal performance. In this paper, we propose Adapters Selector (AS): a novel framework for better integrating usage of multiple adapters by training a middleman adapter to select the appropriate adapter for inference. Our approach utilizes PEFT to train a selector that determines which input content corresponds to which task in which domain, and subsequently selects the homologous adapter. By the way, The AS has developed the capability to execute cross-domain multi-tasks effectively through the utilization of a compact model in combination with multiple LoRA modules. Our code is publicly available.
BibTeX
@inproceedings{tian2025adapters,
  author = {Yimin Tian and Bolin Zhang and Zhiying Tu and Dianhui Chu},
  title = {{Adapters Selector: Cross-domains and Multi-tasks LoRA Modules Integration Usage Method}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=v2iX5BPdUS},
}
Figure