Selecting Informative Contexts Improves Language Model Finetuning

Richard Antonello, Javier Turek, Alexander Huth

Abstract

Language model fine-tuning is essential for modern natural language processing, but is computationally expensive and time-consuming. Further, the effectiveness of fine-tuning is limited by the inclusion of training examples that negatively affect performance. Here we present a general fine-tuning method that we call information gain filtration for improving the overall training efficiency and final performance of language model fine-tuning. We define the information gain of an example as the improvement on a test metric after training on that example. A secondary learner is then trained to approximate this quantity. During fine-tuning, this learner selects informative examples and skips uninformative ones. We show that our method has consistent improvement across datasets, fine-tuning tasks, and language model architectures. For example, we achieve a median perplexity of 54.0 on a books dataset compared to 57.3 for standard fine-tuning. We present statistical evidence that offers insight into the improvements of our method over standard fine-tuning. The generality of our method leads us to propose a new paradigm for language model fine-tuning -- we encourage researchers to release pretrained secondary learners on common corpora to promote efficient and effective fine-tuning, thereby improving the performance and reducing the overall energy footprint of language model fine-tuning.

Introduction

Fine-tuning large pre-trained language models (LMs) like GPT-2 or BERT is a cornerstone of modern NLP. However, the process is notoriously unstable and computationally demanding. Recent findings highlight a significant sensitivity to the random order of training data; certain examples can actively degrade model performance. Consequently, practitioners often resort to "best-of-N" random seed searches, which is inefficient and environmentally costly. This work introduces Information Gain Filtration (IGF), a method to proactively identify and prioritize informative training examples, thereby boosting performance while reducing compute requirements.

Information Gain Filtration (IGF)

The core of IGF is the concept of Information Gain (IG), defined as the reduction in perplexity on a held-out validation set resulting from a single backpropagation step on a specific training example. Since calculating true IG for every example is computationally prohibitive, we employ a lightweight secondary learner (e.g., a small CNN or linear regression model) to predict IG directly from text features.

During training, the secondary learner scores each batch. Examples exceeding a predicted IG threshold are used for backpropagation, while others are skipped. To account for the primary model's evolving state, we use Scheduled Thresholding, starting with a strict filter that gradually relaxes as the model converges.

Experimental Results

We evaluated IGF across diverse datasets including the Books Corpus, WikiText-103, and Reddit comments, as well as downstream tasks like sentiment classification (SST-2).

Training Efficiency and Convergence

IGF significantly accelerates convergence. On the Books dataset, IGF reached lower perplexity levels much faster than standard training. Specifically, IGF achieved a median perplexity of 54.0, a substantial improvement over the 57.3 achieved by standard fine-tuning.

IGF demonstrates faster convergence and superior final performance compared to standard fine-tuning on the Books dataset.

Generality Across Architectures

The benefits of IGF are not limited to a single model or task. IGF consistently outperforms standard fine-tuning across different versions of GPT-2 and BERT, improving both language modeling perplexity and classification accuracy.

IGF consistently improves performance across various model architectures (GPT-2, BERT) and tasks (Language Modeling, SST-2 Classification).

Analysis and Insights

Our analysis revealed several key properties of the fine-tuning process:

  • Early Batch Influence: The quality of initial training batches strongly correlates with final model performance. By filtering for high-IG examples early on, IGF places the model on a superior optimization trajectory.
  • Representativeness: The secondary learner tends to favor representative text (e.g., clear dialogue) over uninformative, complex, or atypical sequences.
  • Efficiency: In early training phases, IGF can reduce the required backpropagation steps by 30-40% to reach a target perplexity.

Conclusion

Information Gain Filtration offers a more principled and efficient alternative to brute-force random seed searching. By leveraging lightweight secondary learners, we can significantly reduce the computational and environmental footprint of LM fine-tuning. We propose a new paradigm where researchers release pre-trained secondary learners alongside major corpora, enabling the community to benefit from efficient, high-performance fine-tuning out-of-the-box.

Summary updated

Abstract

Language model fine-tuning is essential for modern natural language processing, but is computationally expensive and time-consuming. Further, the effectiveness of fine-tuning is limited by the inclusion of training examples that negatively affect performance. Here we present a general fine-tuning method that we call information gain filtration for improving the overall training efficiency and final performance of language model fine-tuning. We define the information gain of an example as the improvement on a test metric after training on that example. A secondary learner is then trained to approximate this quantity. During fine-tuning, this learner selects informative examples and skips uninformative ones. We show that our method has consistent improvement across datasets, fine-tuning tasks, and language model architectures. For example, we achieve a median perplexity of 54.0 on a books dataset compared to 57.3 for standard fine-tuning. We present statistical evidence that offers insight into the improvements of our method over standard fine-tuning. The generality of our method leads us to propose a new paradigm for language model fine-tuning -- we encourage researchers to release pretrained secondary learners on common corpora to promote efficient and effective fine-tuning, thereby improving the performance and reducing the overall energy footprint of language model fine-tuning.
BibTeX
@inproceedings{antonello2021selecting,
  author = {Richard Antonello and Javier Turek and Alexander Huth},
  title = {{Selecting Informative Contexts Improves Language Model Finetuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=NR7IlNiIs8},
}
Figure