Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

Josip Jukic, Jan Snajder

Abstract

Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.

Introduction

Fine-tuning Pre-trained Language Models (PLMs) like BERT typically requires substantial labeled datasets to achieve optimal performance. However, in "low-resource" scenarios—such as specialized medical domains or rare languages—obtaining thousands of labels is often cost-prohibitive. To address this, researchers have historically utilized two distinct strategies: Active Learning (AL), which strategically selects the most informative samples for human labeling, and Parameter-Efficient Fine-Tuning (PEFT), which limits the number of trainable weights to prevent overfitting.

Despite the individual success of these approaches, their combined efficacy has remained largely unexamined. This study investigates the interplay between PEFT and AL, exploring whether parameter-efficient methods enhance the selection process of active learning and why they might offer superior performance over traditional Full Fine-Tuning (FFT) in data-constrained environments.

Experimental Framework

Methods and PEFT Techniques

The study compares FFT against four prominent PEFT architectures:

  • Adapters: Small bottleneck layers inserted between existing Transformer layers.
  • Prefix-tuning: Trainable "prefix" vectors added to the hidden states in the attention mechanism.
  • LoRA (Low-Rank Adaptation): Low-rank matrices injected into Transformer layers to approximate weight updates.
  • UniPELT: A unified framework that dynamically activates Adapters, Prefix-tuning, and LoRA via a gating mechanism.

These methods were evaluated across five AL strategies, ranging from uncertainty-based sampling (Maximum Entropy) to diversity-based methods (Core-set) and Discriminative AL. Additionally, the researchers explored the impact of Task-Adaptive Pre-Training (TAPT), where the model is exposed to unlabeled task-specific data prior to the active learning cycle.

Datasets and Evaluation

Experiments were conducted on four text classification benchmarks representing diverse linguistic challenges: SUBJ (subjectivity), TREC (question type), SST (sentiment), and AGN (news classification). To simulate a true low-resource environment, the labeling budget was strictly capped at 1,000 instances.

Key Insights

1. The Superiority of Representation Stability

A central finding of the research is that PEFT’s advantage in low-resource settings is driven by representation stability. By freezing the majority of the base model, PEFT preserves the foundational linguistic knowledge in the early and middle layers. Using Centered Kernel Alignment (CKA), the authors demonstrated that PEFT representations in the first eight layers remain significantly closer to the original pre-trained state than those of FFT. This "anchoring" effect allows the model to adapt to specific tasks without losing generalizable features.

2. Forgetting Dynamics and Sample Selection

The study utilized "forgetting dynamics"—tracking how often a model correctly classifies an instance and subsequently misclassifies it—to understand model behavior. FFT was found to be more prone to instability and catastrophic forgetting. In contrast, PEFT methods tend to select fewer "unforgettable" (overly simple) instances and more "moderately forgettable" (optimally challenging) examples. These moderately challenging samples provide the highest utility for the learning process, explaining why PEFT synergizes so effectively with AL strategies.

3. Performance Gains and RIPL

UniPELT and Prefix-tuning consistently outperformed FFT across all tested datasets. The researchers measured this using Relative Improvement over Passive Learning (RIPL), finding that PEFT methods achieved higher RIPL scores than FFT. This indicates that PEFT is not only better at learning from small data but is also more capable of leveraging the "smart" data choices made by AL algorithms. In several instances, PEFT with only 200–300 labels matched the performance of FFT using significantly larger datasets.

Discussion and Limitations

While the synergy between PEFT and AL is clear, several considerations remain for future research:

  • Task Complexity: This study focused on text classification. The dynamics of PEFT and AL may differ in sequence labeling (NER), summarization, or translation tasks.
  • Model Scaling: The experiments utilized BERT-base. As models scale to billions of parameters, the interaction between AL and methods like LoRA may evolve.
  • Hyperparameter Sensitivity: PEFT methods introduce new hyperparameters (e.g., bottleneck size). In low-resource settings, the lack of a large validation set makes tuning these parameters a significant challenge.

Conclusion

This research demonstrates that PEFT and Active Learning are highly complementary. By maintaining representation stability and focusing on informative, moderately forgettable samples, PEFT architectures like UniPELT and Prefix-tuning provide a robust framework for fine-tuning in low-resource environments. These findings suggest that the combination of parameter efficiency and strategic sampling is essential for the practical deployment of language models where data is scarce.

Summary updated

Abstract

Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.
BibTeX
@inproceedings{jukic2023parameter,
  author = {Josip Jukic and Jan Snajder},
  title = {{Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8VqB2nDIwD},
}
Figure