Abstract
Parameter-efficient fine-tuning (PEFT) has shown its effectiveness in adapting pre-trained language models to downstream tasks while updating only a small number of parameters. Despite this success, most existing methods independently adapt to each task without considering knowledge transfer between tasks and are limited in low-data regimes. To overcome this, we propose Prototype-based HyperAdapter (PHA), a novel framework built on adapter-tuning and hypernetworks. It introduces an instance-dense retriever and a prototypical hypernetwork to generate conditional modules in a sample-efficient manner. PHA achieves comparable performance improvements against existing PEFT methods on multi-task learning and few-shot transfer learning. Most importantly, when available data size is small, our method outperforms strong baselines by a large margin. Extensive empirical experiments demonstrate that PHA strikes a superior trade-off between trainable parameters, accuracy, and sample efficiency.
Introduction
Fine-tuning massive Pre-trained Language Models (PLMs) for every new task is computationally expensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like Adapters or LoRA help by updating only a tiny fraction of weights, they often treat tasks as isolated islands. This leads to two major issues: a lack of knowledge transfer between similar tasks and high data requirements, where models struggle to generalize in "low-data regimes" (e.g., 100 samples per task).
Existing hypernetwork-based approaches attempt to solve this by sharing a central weight-generator, but they often struggle to maintain task-specific separation, leading to poor performance when data is limited. The Prototype-based HyperAdapter (PHA) addresses these limitations by explicitly structuring the task embedding space using contrastive learning.
Methodology
PHA consists of two core components that work in tandem to generate task-specific adapters based on input data prototypes rather than static task IDs.
Instance-dense Retriever (IDR)
Instead of using a static ID for a task, PHA uses a retriever to project input instances into an embedding space. It employs a contrastive loss (InfoNCE) to ensure that instances from the same task cluster together while instances from different tasks are pushed apart. This explicitly structures the task embedding space to prevent "negative transfer," where learning one task accidentally hurts performance on another.
Prototypical HyperNetworks
The system calculates "prototypes"—the center points of these task clusters—which act as keys for the hypernetwork. When the model processes an input, the retriever identifies the corresponding task prototype. The hypernetwork then takes this prototype and a layer-specific embedding to generate the weights for a small Adapter module inserted into the transformer layers. This instance-based approach allows the model to handle unknown tasks during generalization by finding the closest known prototype in the embedding space.
Experimental Results
The researchers evaluated PHA across a broad range of Natural Language Understanding (NLU) benchmarks, including GLUE, SuperGLUE, and various few-shot transfer tasks.
Dominance in Low-Data Regimes
PHA shines brightest when data is scarce. In a 100-sample-per-task setting on GLUE, PHA outperformed standard adapter-tuning by a massive 8.0%. By anchoring the hypernetwork to well-separated prototypes, the model requires far fewer examples to understand the requirements of a specific task compared to baselines like Hyperformer++ or HyperDecoder.
Multi-task Performance and Scalability
On the full GLUE benchmark, PHA achieved an average score of 85.5, outperforming strong baselines like Hyperformer++ (84.7) and HyperDecoder (83.7) while using significantly fewer trainable parameters (616K vs 1.8M for HyperDecoder). Furthermore, the benefits of PHA held steady as the backbone model scaled from T5-Small to T5-Large, suggesting the method is robust for modern, massive architectures.
Conclusion
PHA provides a sample-efficient framework for multi-task tuning by leveraging contrastive learning to define task prototypes. This approach enables superior knowledge transfer and generalization, particularly in low-resource scenarios. Future work may explore "mixture-of-prototypes" approaches to leverage knowledge from multiple related tasks and the application of PHA to other PEFT modules like LoRA or Prefix-tuning.
Summary updated