An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

Introduction

Transfer learning has become the cornerstone of modern Natural Language Processing (NLP). The prevailing paradigm involves pretraining a language model (LM) on a massive corpus and subsequently "fine-tuning" it for specific downstream tasks. However, this transition often triggers catastrophic forgetting, where the model abruptly loses the general linguistic knowledge (syntax, semantics) acquired during pretraining as it adapts to the new task.

While existing solutions like ULMFiT address this through complex multi-stage pipelines involving domain-specific fine-tuning and intricate learning rate schedules, they can be cumbersome to implement. This paper introduces SiATL (Single-step Auxiliary loss Transfer Learning), a streamlined approach that simplifies transfer learning into a single end-to-end step while maintaining high performance and robustness.

Methodology: The SiATL Approach

SiATL avoids the multi-stage complexity of previous methods by integrating an auxiliary objective directly into the fine-tuning process.

Architecture and Pretraining

The process begins with a standard word-level LSTM language model pretrained on a large-scale corpus (e.g., 20 million Twitter messages). To adapt this for classification, the pretrained weights are transferred to a new architecture that includes a task-specific recurrent layer (LSTM with self-attention) and a final classification layer.

Comparison of the base language model (left) and the SiATL classification model with auxiliary loss (right). Red dots indicate transferred weights, while grey dots represent randomly initialized components.

Joint Loss and Exponential Decay

The core innovation of SiATL is the use of a joint loss function: $$L = L_{task} + \gamma L_{LM}$$ Here, $L_{task}$ represents the standard classification loss, while $L_{LM}$ is an auxiliary language modeling loss. This auxiliary loss acts as a powerful regularizer, forcing the model to retain its language modeling capabilities while learning the target task.

To allow the model to eventually specialize in the downstream task, the weight of the auxiliary loss ($\gamma$) is not static. It starts at a high value to protect pretrained weights and decays exponentially throughout training, eventually allowing the model to focus entirely on the target task.

Sequential Unfreezing

To further stabilize training, the authors employ sequential unfreezing. This prevents the gradients from randomly initialized layers from "corrupting" the pretrained weights early in the process. The model is unfrozen in stages: first the task-specific layers, then the hidden layers, and finally the embedding layer.

Experimental Results

The effectiveness of SiATL was evaluated across five diverse datasets covering sentiment analysis (Sent17), irony detection (Irony18), emotion recognition (PsychExp), and sarcasm detection (SCv1, SCv2).

Performance and Data Efficiency

SiATL consistently outperformed ULMFiT across four of the five datasets. A key strength of the approach is its performance in low-resource scenarios. By leveraging the auxiliary loss, the model remains grounded in general language patterns even when target data is scarce. For instance, on the SCv1 dataset, SiATL provided a massive 7% boost in F1-score compared to a model initialized with the same weights but without the auxiliary loss.

F1-score performance across different datasets as a function of the number of training examples, highlighting SiATL's robustness in low-resource settings.

As shown in the figure above, SiATL maintains competitive performance even when training sets are reduced to fewer than 1,000 examples, significantly outperforming more complex alternatives in these "small data" regimes.

Conclusion

SiATL demonstrates that effective transfer learning in NLP does not require overly complex pipelines. By using a simple auxiliary loss with exponential decay and sequential unfreezing, the method achieves state-of-the-art results while being significantly easier to implement than multi-stage approaches like ULMFiT. Future work may involve scaling the vocabulary using sub-word units and applying this auxiliary loss framework to bidirectional transformer architectures.

Summary updated

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.
BibTeX
@inproceedings{chronopoulou2019embarrassingly,
  author = {Alexandra Chronopoulou and Christos Baziotis and Alexandros Potamianos},
  title = {{An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1213},
}
Figure