Abstract
Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting pre-trained language models to downstream tasks while only updating a small number of parameters. Despite this success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited in low-data regimes. To overcome this issue, we propose Prototype-based HyperAdapter (PHA), a novel framework built on adapter-tuning and hypernetworks. It introduces an instance-dense retriever and a prototypical hypernetwork to generate conditional modules in a sample-efficient manner. This leads to comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. More importantly, when the available data size gets smaller, our method outperforms other strong baselines by a large margin. Based on our extensive empirical experiments across various datasets, we demonstrate that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency.
Introduction
Fine-tuning massive pre-trained language models (PLMs) for every new downstream task is computationally expensive and storage-intensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like adapters have mitigated storage issues by updating only a fraction of the total parameters, they often treat tasks as isolated silos. This independence prevents models from sharing knowledge across related tasks—such as sentiment analysis and linguistic acceptability—and often leads to poor performance in low-data regimes where training examples are scarce.
Existing hypernetwork approaches attempt to address this by generating adapter weights dynamically. However, these methods frequently struggle with task-level representation and overfit when data is limited. The Prototype-based HyperAdapter (PHA) addresses these challenges by bridging the gap between instance-level data and task-level knowledge through a shared hypernetwork and learned task prototypes.
The PHA Framework
PHA utilizes a three-part architecture designed to maximize knowledge transfer and sample efficiency:
- Instance-dense Retriever (IDR): This component projects input instances into a shared embedding space. It is trained using a contrastive loss (InfoNCE), which encourages instances from the same task to cluster together while pushing instances from different tasks apart. This provides critical "side information" that helps the model understand task structures even with limited labels.
- Prototypical HyperNetworks: Rather than learning static, arbitrary embeddings for each task, PHA calculates a "prototype"—the central point of all instance embeddings for a given task. These prototypes serve as stable anchors for the hypernetwork.
- HyperAdapter Generation: A shared hypernetwork takes these prototypes and layer-specific embeddings as input to generate weights for parallel adapters. These adapters are then inserted into the PLM's Feed-Forward Networks (FFN).
When encountering a new task, the retriever identifies the most similar existing task prototype. This retrieved knowledge allows the hypernetwork to generate effective adapter weights for the new task even with very few training examples, enabling robust few-shot generalization.
Experimental Evaluation
The researchers evaluated PHA across 13 diverse NLP datasets, including the GLUE and SuperGLUE benchmarks. To test generalization, the SciTail dataset (science question answering) was used for few-shot transfer learning with as few as 4, 16, or 32 examples.
Key Results
- Efficiency and Accuracy: PHA uses only 616K trainable parameters—approximately 0.28% of a T5-Base model. Despite this small footprint, it achieved an average score of 85.5 on GLUE, outperforming both full fine-tuning (84.9) and standard adapter methods (84.5).
- Low-Data Performance: In extreme low-resource settings (100 samples per task), PHA outperformed standard adapter tuning by a significant 8.0% margin.
- Few-Shot Generalization: On the SciTail dataset with 32 samples, PHA achieved 88.6% accuracy, surpassing the next best baseline (MPT) by 2.3%.
- Scalability: The prototypical strategy proved effective across different model scales, from T5-Small (60M) to T5-Large (770M).
Discussion and Future Work
PHA demonstrates that anchoring task representations to data-driven prototypes prevents the "intertwined" information problem common in previous hypernetworks. This precision allows for functional adapter generation even in zero-shot or near-zero-shot scenarios by recognizing similarities between new inputs and known task prototypes.
However, the current implementation selects only the single most relevant prototype for a new task. Future work may explore using weighted combinations of prototypes to better represent tasks that blend multiple domains. Additionally, while the current study focused on parallel adapters, the prototypical approach could be extended to other PEFT architectures like LoRA or Prefix-tuning to further enhance multi-task efficiency.
Summary updated