Abstract
Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL) to train jointly on a supplementary task and the target task (pairwise MTL), or simply using MTL to train jointly on all available datasets (MTL-ALL). In this work, we compare all three TL methods in a comprehensive analysis on the GLUE dataset suite. We find that there is a simple heuristic for when to use one of these techniques over the other: pairwise MTL is better than STILTs when the target task has fewer instances than the supporting task and vice versa. We show that this holds true in more than 92% of applicable cases on the GLUE dataset and validate this hypothesis with experiments varying dataset size. The simplicity and effectiveness of this heuristic is surprising and warrants additional exploration by the TL community. Furthermore, we find that MTL-ALL is worse than the pairwise methods in almost every case. We hope this study will aid others as they choose between TL methods for NLP tasks.
Introduction
In modern NLP, practitioners rarely train models from scratch. The standard workflow involves taking a pre-trained encoder, such as BERT or RoBERTa, and fine-tuning it on a specific target task. Often, researchers have access to "supporting" datasets—additional labeled tasks that can potentially improve the model's performance on the target task.
However, the best way to integrate this extra data remains a subject of debate. Two primary strategies dominate the field:
- STILTs (Supplementary Training on Intermediate Labeled-data Tasks): Sequential training where the model learns the supporting task first, then the target task.
- Multi-Task Learning (MTL): Simultaneous training where the model learns both tasks at once.
This paper provides a comprehensive head-to-head comparison of these strategies, revealing a surprisingly simple "size heuristic" that predicts the optimal method with high accuracy.
Methodology
The researchers conducted a massive evaluation using DistilRoBERTa and BERT across the GLUE (General Language Understanding Evaluation) benchmark. GLUE provides a diverse range of tasks and dataset sizes, from the massive MNLI (nearly 400,000 instances) to the tiny WNLI (635 instances).
The study compared three main configurations:
- STILTs: Sequential training (Supporting → Target).
- Pairwise MTL: Joint training on exactly two tasks (Supporting + Target).
- MTL-ALL: Joint training on all nine GLUE tasks simultaneously.
To ensure robustness, each configuration was trained with five different random seeds, and statistical t-tests were employed to confirm the significance of the performance differences.
The Size Heuristic
The most significant insight from this research is that the relative size of the datasets is the primary driver of success. The authors propose a simple rule of thumb:
- Use Pairwise MTL if the target task is smaller than the supporting task.
- Use STILTs if the target task is larger than the supporting task.
This heuristic correctly predicted the superior training method in 92.5% of statistically significant cases (49 out of 53) across the GLUE benchmark. While many practitioners focus on "task similarity" (e.g., training a sentiment model on other sentiment data), these results suggest that for encoder-based transfer learning, raw data volume is a much more reliable predictor of which method to use.
Key Findings
The "Crossover" Point
To validate the size heuristic, the authors performed controlled experiments by artificially subsampling datasets. They demonstrated that as a supporting task grows in size relative to a fixed target task, the performance of MTL steadily climbs until it eventually overtakes STILTs. This "crossover" confirms that the performance advantage is directly tied to the ratio of dataset sizes.
The Failure of MTL-ALL
One might assume that training on all available data (the "kitchen sink" approach) would produce the strongest model. However, the study found that MTL-ALL was almost universally the worst method. Jointly training on too many tasks appears to dilute the model's focus on the specific target task. Pairwise methods (either STILTs or Pairwise MTL) consistently outperformed the MTL-ALL approach.
Performance Gains
Choosing the correct method based on the size heuristic is not just a marginal improvement. In several cases, it led to substantial performance jumps—up to a 9-point difference on tasks like WNLI or STS-B. These findings held true across different architectures (DistilRoBERTa and BERT), suggesting a fundamental property of how these encoders learn.
Conclusion and Limitations
The size heuristic offers a practical, low-cost way for researchers to choose a transfer learning strategy without exhaustive experimentation. However, it is not a universal law; the authors noted a few exceptions (primarily involving the MRPC dataset) where task-specific nuances outweighed dataset size.
Future work is required to determine if these rules apply to decoder-only models (like GPT) or encoder-decoder architectures (like T5). Additionally, the theoretical reason for this phenomenon—whether MTL acts as a regularizer for small tasks or STILTs provides a better starting initialization for large ones—remains an open question for the transfer learning community.
Summary updated