Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings

Josip Jukic, Jan Snajder

Abstract

Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.

Introduction

Training high-performance Natural Language Processing (NLP) models typically requires massive amounts of labeled data—a luxury many "low-resource" languages and specialized domains simply don't have. Two popular strategies exist to mitigate this labeling bottleneck. First, Active Learning (AL) strategically selects the most "informative" examples for human labeling rather than picking data at random. Second, Parameter-Efficient Fine-Tuning (PEFT) updates only a tiny fraction of a model's parameters (e.g., using adapters) instead of the entire model. While both fields are well-studied independently, their intersection has remained largely uncharted territory. Researchers previously did not know if the "informativeness" of a data point selected for a Full Fine-Tuning (FFT) model would still be relevant for a PEFT model, or if the inherent instability of FFT in low-data regimes was sabotaging the potential of Active Learning.

Methodology

The authors conducted an extensive empirical study using BERT-base as the backbone model. They compared four PEFT techniques against Full Fine-Tuning (FFT):

  • Adapters: Small bottleneck layers inserted between Transformer blocks.
  • Prefix-tuning: Trainable tensors added to the hidden states of the attention mechanism.
  • LoRA (Low-Rank Adaptation): Injecting trainable low-rank matrices into the model layers.
  • UniPELT: A unified approach that combines the three methods above using a gating mechanism.

These methods were evaluated across four text classification benchmarks: SUBJ (subjectivity analysis), TREC (question type classification), SST (sentiment analysis), and AGN (topic categorization). Five AL sampling strategies were tested, including Random (RND), Maximum Entropy (ENT), and Core-set (CS). Additionally, the study examined the impact of Task-Adaptive Pre-Training (TAPT)—unsupervised masked language modeling on the target domain's unlabeled text before active learning begins.

Results and Analysis

PEFT Robustness in Low-Data Regimes

The empirical results demonstrate that PEFT is not just a way to save compute; it is fundamentally more robust in low-resource settings. FFT often exhibits significant "instability" when trained on tiny datasets (100–300 samples), whereas PEFT methods provide a much smoother learning curve. Specifically, UniPELT and Prefix-tuning emerged as the top performers, consistently outperforming FFT across almost all datasets and AL strategies.

F1 scores across SUBJ, TREC, SST, and AGN datasets as labeled data increases, comparing various PEFT methods against full fine-tuning.

The performance gap was most dramatic at the very start of the process (the first 100 labels), where PEFT models achieved F1 scores 5–10 points higher than FFT. This suggests that PEFT methods are better suited for the initial "cold start" phase of active learning.

Synergy with Active Learning

The study found a powerful synergy between PEFT and Active Learning. Using the Relative Improvement over Passive Learning (RIPL) metric, the authors demonstrated that the performance gains from using AL are significantly higher when paired with PEFT than with FFT. This indicates that the stable representations of PEFT models allow AL algorithms to more effectively identify and learn from informative samples.

Relative Improvement over Passive Learning (RIPL) across datasets, highlighting the superior synergy between AL and PEFT compared to FFT.

Representation Stability and Forgetting Dynamics

Analysis of the model's internal dynamics provides a reason for this success. Using Centered Kernel Alignment (CKA), the authors found that PEFT preserves the representations of the early and middle layers of the pre-trained model much better than FFT. By keeping this foundational knowledge intact, the model can focus its limited learning capacity on task-specific features in the final layers. Furthermore, PEFT models are better at learning and retaining "moderately forgettable" examples—the informative data points that are most beneficial for AL.

Conclusion

The combination of TAPT, PEFT (particularly UniPELT or Prefix-tuning), and entropy-based AL constitutes a highly effective strategy for low-resource text classification. PEFT's ability to maintain stable internal representations and resist forgetting makes it an ideal partner for Active Learning. Future research should extend these findings to larger models and more complex generative tasks.

Summary updated

Abstract

Pre-trained language models (PLMs) have ignited a surge in demand for effective fine-tuning techniques, particularly in low-resource domains and languages. Active learning (AL), a set of algorithms designed to decrease labeling costs by minimizing label complexity, has shown promise in confronting the labeling bottleneck. In parallel, adapter modules designed for parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in low-resource settings. However, the interplay between AL and adapter-based PEFT remains unexplored. We present an empirical study of PEFT behavior with AL in low-resource settings for text classification tasks. Our findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-resource settings and demonstrate that this advantage persists in AL setups. We further examine the properties of PEFT and FFT through the lens of forgetting dynamics and instance-level representations, where we find that PEFT yields more stable representations of early and middle layers compared to FFT. Our research underscores the synergistic potential of AL and PEFT in low-resource settings, paving the way for advancements in efficient and effective fine-tuning.
BibTeX
@inproceedings{jukic2023parameter,
  author = {Josip Jukic and Jan Snajder},
  title = {{Parameter-Efficient Language Model Tuning with Active Learning in Low-Resource Settings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=8VqB2nDIwD},
}
Figure