Abstract
Adapting Pre-trained Language Models (PLMs) to downstream tasks with extremely limited data remains a significant challenge. Traditional fine-tuning often leads to overfitting in few-shot scenarios, while prompt tuning requires intensive manual effort. This paper introduces AMAL (Adapter-by-MetA-Learning), a novel framework that leverages meta-learning and low-rank matrix decomposition to generate task-specific adapters. By learning a shared pool of meta-knowledge (adapter kernels) and a task-specific pooling mechanism, AMAL achieves state-of-the-art performance in few-shot text classification with high parameter efficiency.
Introduction
While PLMs like BERT and RoBERTa have revolutionized NLP, their effectiveness in the few-shot regime (e.g., 1 or 5 examples per class) is often hampered by the instability of standard fine-tuning. Existing parameter-efficient fine-tuning (PEFT) methods, such as adapters, reduce the number of trainable parameters but still struggle when data is scarce. Prompt tuning offers an alternative but is highly sensitive to the choice of prompts, necessitating labor-intensive engineering.
AMAL addresses these issues by automating the adaptation process through a meta-learning lens. It treats new tasks not as isolated problems but as specific combinations of universal adaptation patterns stored in a shared meta-knowledge base.
Methodology
The core innovation of AMAL is its meta-learning-driven low-rank adapter pooling mechanism.
Low-Rank Decomposition
Drawing on the observation that PLMs possess a low intrinsic dimension, AMAL decomposes the adapter's projection matrix $P$ into three distinct components: $$P = U \times E \times V^T$$ Where:
- $U$ and $V$ (Adapter Kernels): These matrices are shared across all tasks and capture general, reusable adaptation patterns.
- $E$ (Task-Specific Pooler): A diagonal matrix that determines the optimal combination of kernels for a specific task.
Meta-Learning Framework
AMAL employs a bi-level optimization strategy:
- Inner Loop: For a given task, a Relation Network encodes the [CLS] tokens of support examples into a latent task embedding $z$. A decoder then generates the task-specific matrix $E$.
- Outer Loop: The shared kernels ($U, V$) and the encoder-decoder parameters are updated across a variety of tasks to ensure robust meta-knowledge. A near-orthogonality penalty is applied to $U$ and $V$ to ensure the kernels capture diverse, non-redundant information.
Experimental Evaluation
The authors evaluated AMAL across five diverse text classification benchmarks: Amazon (product reviews), Huffpost (news headlines), RCV-1 (Reuters stories), Reuters-21578, and 20 Newsgroups. Experiments focused on 5-way 1-shot and 5-way 5-shot settings, where the model must learn to classify five categories given only one or five examples per category.
Results and Discussion
AMAL demonstrates a substantial performance leap over existing baselines. On average, it outperformed specialized few-shot methods by 27.69% in 1-shot settings and 22.03% in 5-shot settings. The method proved versatile, showing consistent gains across various architectures including BERT, RoBERTa, and DeBERTa.
Efficiency and Layer Sensitivity
A key finding is the sensitivity of the model to the placement of adapters. The authors observed that performance gains are most significant when adapters are placed in the top-most layers of the PLM.
As shown in the figure above, adding adapters beyond the first 6-7 layers (of a 12-layer model) yields diminishing returns. This allows AMAL to be even more efficient by only equipping the final layers. In terms of total parameters, AMAL uses only 0.595M fine-tuning parameters for a BERT-base model, nearly half of the 1.23M required by standard Houlsby adapters, while delivering significantly higher accuracy.
Conclusion
AMAL provides a robust and efficient solution for few-shot adaptation of PLMs. By framing adaptation as a meta-learning problem over low-rank kernels, it bypasses the need for manual prompt engineering and mitigates the instability of traditional fine-tuning. While currently optimized for bidirectional models, future work includes extending this framework to generative tasks and causal (GPT-style) architectures.
Summary updated