Introduction
Transfer learning has become the cornerstone of modern Natural Language Processing (NLP). The prevailing paradigm involves pretraining a language model (LM) on a massive corpus and subsequently "fine-tuning" it for specific downstream tasks. However, this transition often triggers catastrophic forgetting, where the model abruptly loses the general linguistic knowledge (syntax, semantics) acquired during pretraining as it adapts to the new task.
While existing solutions like ULMFiT address this through complex multi-stage pipelines involving domain-specific fine-tuning and intricate learning rate schedules, they can be cumbersome to implement. This paper introduces SiATL (Single-step Auxiliary loss Transfer Learning), a streamlined approach that simplifies transfer learning into a single end-to-end step while maintaining high performance and robustness.
Methodology: The SiATL Approach
SiATL avoids the multi-stage complexity of previous methods by integrating an auxiliary objective directly into the fine-tuning process.
Architecture and Pretraining
The process begins with a standard word-level LSTM language model pretrained on a large-scale corpus (e.g., 20 million Twitter messages). To adapt this for classification, the pretrained weights are transferred to a new architecture that includes a task-specific recurrent layer (LSTM with self-attention) and a final classification layer.
Joint Loss and Exponential Decay
The core innovation of SiATL is the use of a joint loss function: $$L = L_{task} + \gamma L_{LM}$$ Here, $L_{task}$ represents the standard classification loss, while $L_{LM}$ is an auxiliary language modeling loss. This auxiliary loss acts as a powerful regularizer, forcing the model to retain its language modeling capabilities while learning the target task.
To allow the model to eventually specialize in the downstream task, the weight of the auxiliary loss ($\gamma$) is not static. It starts at a high value to protect pretrained weights and decays exponentially throughout training, eventually allowing the model to focus entirely on the target task.
Sequential Unfreezing
To further stabilize training, the authors employ sequential unfreezing. This prevents the gradients from randomly initialized layers from "corrupting" the pretrained weights early in the process. The model is unfrozen in stages: first the task-specific layers, then the hidden layers, and finally the embedding layer.
Experimental Results
The effectiveness of SiATL was evaluated across five diverse datasets covering sentiment analysis (Sent17), irony detection (Irony18), emotion recognition (PsychExp), and sarcasm detection (SCv1, SCv2).
Performance and Data Efficiency
SiATL consistently outperformed ULMFiT across four of the five datasets. A key strength of the approach is its performance in low-resource scenarios. By leveraging the auxiliary loss, the model remains grounded in general language patterns even when target data is scarce. For instance, on the SCv1 dataset, SiATL provided a massive 7% boost in F1-score compared to a model initialized with the same weights but without the auxiliary loss.
As shown in the figure above, SiATL maintains competitive performance even when training sets are reduced to fewer than 1,000 examples, significantly outperforming more complex alternatives in these "small data" regimes.
Conclusion
SiATL demonstrates that effective transfer learning in NLP does not require overly complex pipelines. By using a simple auxiliary loss with exponential decay and sequential unfreezing, the method achieves state-of-the-art results while being significantly easier to implement than multi-stage approaches like ULMFiT. Future work may involve scaling the vocabulary using sub-word units and applying this auxiliary loss framework to bidirectional transformer architectures.
Summary updated