When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning

Orion Weller, Kevin D. Seppi, Matt Gardner

Introduction

In modern Natural Language Processing (NLP), the standard practice involves fine-tuning a pre-trained model, such as RoBERTa, on a specific target task. Performance can often be enhanced by incorporating "supporting" supervised datasets. However, practitioners face a strategic choice: should they use Intermediate Fine-Tuning (STILTs)—training on the supporting task before the target task—or Multi-Task Learning (MTL)—training on both tasks simultaneously? This paper presents a systematic empirical study that establishes a simple "size heuristic" to determine the optimal strategy based on dataset scale.

Methodology

The authors conducted a massive empirical study comparing three primary transfer learning (TL) strategies to move beyond the "vibe-based" selection process common in the field. They utilized DistilRoBERTa as their primary model, validating results with BERT. The study utilized nine diverse tasks from the GLUE benchmark, ranging from large-scale (MNLI, >390,000 instances) to small-scale (WNLI, 635 instances).

The strategies compared were:

  1. STILTs: Sequential fine-tuning (Pre-train $\rightarrow$ Supporting Task $\rightarrow$ Target Task).
  2. Pairwise MTL: Joint training on two tasks (Pre-train $\rightarrow$ [Supporting + Target]).
  3. MTLAll: Joint training on all available tasks (Pre-train $\rightarrow$ [All Tasks]).

A key technical detail was the implementation of a dynamic sampling strategy for MTL to ensure tasks were balanced during training. To ensure statistical rigor, every configuration was run across five different random seeds, with t-tests used to confirm the reliability of the performance differences.

The Size Heuristic

The researchers discovered that the relative size of the target and supporting datasets is the most reliable predictor of transfer success. This led to the formulation of the Size Heuristic:

  • Target < Supporting: Use MTL. The larger supporting task provides essential regularization, preventing the model from overfitting to the limited data of the target task.
  • Target > Supporting: Use STILTs. The supporting task serves as a beneficial "warm start" initialization, while the larger target dataset is sufficient for effective final fine-tuning.
Figure 1: Heatmap of performance differences between MTL and STILTs. Blue cells indicate MTL is superior; green cells indicate STILTs is better. The distribution clearly follows the relative task sizes.

This heuristic correctly identified the superior method in 92.5% of statistically significant cases. Controlled experiments further revealed a "crossover point": as the supporting task grows larger relative to the target, the performance advantage shifts decisively from STILTs to MTL.

Empirical Results

The study demonstrates that targeted pairwise transfer guided by the size heuristic significantly outperforms the common practice of training on all available tasks (MTLAll). While it is tempting to throw every dataset into one big multi-task hopper, the authors found that MTLAll was almost universally the worst-performing method, likely due to gradient interference between unrelated tasks.

Figure 6: Performance comparison of transfer learning approaches. The Size Heuristic (Avg. S.H.) achieves the highest average score among practical methods, significantly outperforming the MTLAll baseline.

Key findings include:

  • Predictive Accuracy: The heuristic correctly predicted the better method in 49 out of 53 significant cases.
  • Performance Gains: Implementing the size heuristic resulted in an average GLUE score of 78.3, a substantial improvement over the 73.3 achieved by MTLAll.
  • Robustness: These patterns held across different model architectures, suggesting a fundamental characteristic of how encoders learn from multiple data sources.

Limitations and Future Work

The heuristic encountered an exception with the MRPC dataset, indicating that task similarity or specific linguistic features can occasionally override the influence of dataset size. Additionally, while the empirical evidence is compelling, the underlying theoretical mechanisms—such as the interplay between catastrophic forgetting in sequential training and gradient interference in joint training—remain an area for future research. Extending this heuristic to optimize clusters of three or more tasks also presents a promising direction for further study.

Summary updated

Abstract

Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL) to train jointly on a supplementary task and the target task (pairwise MTL), or simply using MTL to train jointly on all available datasets (MTL-ALL). In this work, we compare all three TL methods in a comprehensive analysis on the GLUE dataset suite. We find that there is a simple heuristic for when to use one of these techniques over the other: pairwise MTL is better than STILTs when the target task has fewer instances than the supporting task and vice versa. We show that this holds true in more than 92% of applicable cases on the GLUE dataset and validate this hypothesis with experiments varying dataset size. The simplicity and effectiveness of this heuristic is surprising and warrants additional exploration by the TL community. Furthermore, we find that MTL-ALL is worse than the pairwise methods in almost every case. We hope this study will aid others as they choose between TL methods for NLP tasks.
BibTeX
@inproceedings{weller2022when,
  author = {Orion Weller and Kevin D. Seppi and Matt Gardner},
  title = {{When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.30},
}
Figure