Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning

Hao Zhao, Jie Fu, Zhaofeng He

Abstract

Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting pre-trained language models to downstream tasks while updating only a small number of parameters. Despite this success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited in low-data regimes. To overcome this, we propose Prototype-based HyperAdapter (PHA), a novel framework built on adapter-tuning and hypernetworks. It introduces an instance-dense retriever and a prototypical hypernetwork to generate conditional modules in a sample-efficient manner. PHA achieves comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. Most importantly, when available data size is small, our method outperforms strong baselines by a large margin. Extensive empirical experiments demonstrate that PHA strikes a superior trade-off between trainable parameters, accuracy, and sample efficiency.

Introduction

Fine-tuning massive Pre-trained Language Models (PLMs) for every new task is computationally expensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like Adapters or LoRA help by updating only a tiny fraction of weights, they often treat tasks as isolated islands. This leads to two major issues: a lack of knowledge transfer between similar tasks and high data requirements, where models struggle to generalize in "low-data regimes" (e.g., 100 samples per task).

Existing hypernetwork-based approaches attempt to solve this by sharing a central weight-generator, but they often struggle to maintain task-specific separation, leading to poor performance when data is limited. The Prototype-based HyperAdapter (PHA) addresses these limitations by explicitly structuring the task embedding space using contrastive learning.

Methodology

PHA consists of two core components that work in tandem to generate task-specific adapters based on input data prototypes rather than static task IDs.

The PHA architecture utilizes an Instance-dense Retriever to map inputs to an embedding space, where a Prototypical HyperNetwork generates adapter weights based on task prototypes.

Instance-dense Retriever (IDR)

Instead of using a static ID for a task, PHA uses a retriever to project input instances into an embedding space. It employs a contrastive loss (InfoNCE) to ensure that instances from the same task cluster together while instances from different tasks are pushed apart. This explicitly structures the task embedding space to prevent "negative transfer," where learning one task accidentally hurts performance on another.

Prototypical HyperNetworks

The system calculates "prototypes"—the center points of these task clusters—which act as keys for the hypernetwork. When the model processes an input, the retriever identifies the corresponding task prototype. The hypernetwork then takes this prototype and a layer-specific embedding to generate the weights for a small Adapter module inserted into the transformer layers. This instance-based approach allows the model to handle unknown tasks during generalization by finding the closest known prototype in the embedding space.

Experimental Results

The researchers evaluated PHA across a broad range of Natural Language Understanding (NLU) benchmarks, including GLUE, SuperGLUE, and various few-shot transfer tasks.

Accuracy comparison across different sample sizes per task. PHA (purple line) consistently outperforms baselines, especially in extremely low-data regimes (e.g., 100 samples/task).

Dominance in Low-Data Regimes

PHA shines brightest when data is scarce. In a 100-sample-per-task setting on GLUE, PHA outperformed standard adapter-tuning by a massive 8.0%. By anchoring the hypernetwork to well-separated prototypes, the model requires far fewer examples to understand the requirements of a specific task compared to baselines like Hyperformer++ or HyperDecoder.

Multi-task Performance and Scalability

On the full GLUE benchmark, PHA achieved an average score of 85.5, outperforming strong baselines like Hyperformer++ (84.7) and HyperDecoder (83.7) while using significantly fewer trainable parameters (616K vs 1.8M for HyperDecoder). Furthermore, the benefits of PHA held steady as the backbone model scaled from T5-Small to T5-Large, suggesting the method is robust for modern, massive architectures.

Conclusion

PHA provides a sample-efficient framework for multi-task tuning by leveraging contrastive learning to define task prototypes. This approach enables superior knowledge transfer and generalization, particularly in low-resource scenarios. Future work may explore "mixture-of-prototypes" approaches to leverage knowledge from multiple related tasks and the application of PHA to other PEFT modules like LoRA or Prefix-tuning.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting the pre-trained language models to downstream tasks while only updating a small number of parameters. Despite the success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited to low-data regimes. To overcome this issue, we propose Prototype-based HyperAdapter (PHA), a novel framework built on the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and a prototypical hypernetwork to generate the conditional modules in a sample-efficient manner. This leads to comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. More importantly, when the available data size gets smaller, our method outperforms other strong baselines by a large margin. Based on our extensive empirical experiments across various datasets, we demonstrate that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency. Our code is publicly available at https://github.com/Bumble666/PHA
BibTeX
@inproceedings{zhao2023prototype,
  author = {Hao Zhao and Jie Fu and Zhaofeng He},
  title = {{Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=iammae3CbG},
}
Figure