An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

Introduction

Transfer learning in Natural Language Processing (NLP) typically involves pretraining a Language Model (LM) on a massive corpus and then fine-tuning it for a specific downstream task. However, this standard pipeline often suffers from catastrophic forgetting, where the model loses general linguistic knowledge while adapting to the nuances of a new, often smaller, dataset.

Existing solutions like ULMFiT or BERT address this through complex learning rate schedules or massive computational resources. This paper introduces SiATL (Single-step Auxiliary loss Transfer Learning), a conceptually simple framework that prevents forgetting by maintaining the original language modeling objective during the fine-tuning phase.

Methodology

The SiATL framework consists of four primary components designed for efficiency and performance:

  1. Pretraining: An LSTM-based language model is trained on a large, domain-relevant corpus (e.g., Twitter).
  2. Auxiliary Loss ($L_{LM}$): The core innovation is the inclusion of the original language modeling objective in the final loss function: $$L = L_{task} + \gamma L_{LM}$$ The parameter $\gamma$ weights the importance of retaining pretrained knowledge.

  3. Exponential Decay of $\gamma$: To allow the model to eventually specialize in the target task, $\gamma$ is decayed exponentially over training epochs.

  4. Sequential Unfreezing: The model is trained by gradually "unfreezing" layers, starting from randomly initialized task-specific layers and moving down to the pretrained LSTM layers.
The SiATL architecture showing the transition from pretraining (left) to transfer learning with an auxiliary loss (right).

Experimental Setup

The model was pretrained on 20 million English Twitter messages. Evaluation was conducted across several affective and text classification benchmarks, including sentiment analysis (Sent17), emotion recognition (PsychExp), irony detection (Irony18), and sarcasm detection (SCv1 & SCv2).

Results and Discussion

SiATL consistently outperforms more complex models like ULMFiT across all benchmark datasets. A key finding is the effectiveness of the LM loss as a regularizer, which prevents overfitting on small downstream datasets.

F1-score performance across different datasets as the number of training examples increases, demonstrating the model's robustness in low-data scenarios.

On the SCv1 dataset, the smallest tested, the auxiliary loss provided a 7% performance boost over standard fine-tuning. Furthermore, the method is highly efficient, requiring only a single GPU for training, unlike many modern transformer-based architectures. This simplicity allows for competitive performance without industrial-scale infrastructure.

Conclusion

SiATL demonstrates that a simple auxiliary loss and smart unfreezing strategy can achieve state-of-the-art results without the complexity of modern transfer learning pipelines. Future work will focus on addressing vocabulary constraints (OOV words) and automating the sequential unfreezing process to further simplify deployment.

Summary updated

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.
BibTeX
@inproceedings{chronopoulou2019embarrassingly,
  author = {Alexandra Chronopoulou and Christos Baziotis and Alexandros Potamianos},
  title = {{An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1213},
}
Figure