Introduction
In modern Natural Language Processing (NLP), the standard practice involves fine-tuning a pre-trained model, such as RoBERTa, on a specific target task. Performance can often be enhanced by incorporating "supporting" supervised datasets. However, practitioners face a strategic choice: should they use Intermediate Fine-Tuning (STILTs)—training on the supporting task before the target task—or Multi-Task Learning (MTL)—training on both tasks simultaneously? This paper presents a systematic empirical study that establishes a simple "size heuristic" to determine the optimal strategy based on dataset scale.
Methodology
The authors conducted a massive empirical study comparing three primary transfer learning (TL) strategies to move beyond the "vibe-based" selection process common in the field. They utilized DistilRoBERTa as their primary model, validating results with BERT. The study utilized nine diverse tasks from the GLUE benchmark, ranging from large-scale (MNLI, >390,000 instances) to small-scale (WNLI, 635 instances).
The strategies compared were:
- STILTs: Sequential fine-tuning (Pre-train $\rightarrow$ Supporting Task $\rightarrow$ Target Task).
- Pairwise MTL: Joint training on two tasks (Pre-train $\rightarrow$ [Supporting + Target]).
- MTLAll: Joint training on all available tasks (Pre-train $\rightarrow$ [All Tasks]).
A key technical detail was the implementation of a dynamic sampling strategy for MTL to ensure tasks were balanced during training. To ensure statistical rigor, every configuration was run across five different random seeds, with t-tests used to confirm the reliability of the performance differences.
The Size Heuristic
The researchers discovered that the relative size of the target and supporting datasets is the most reliable predictor of transfer success. This led to the formulation of the Size Heuristic:
- Target < Supporting: Use MTL. The larger supporting task provides essential regularization, preventing the model from overfitting to the limited data of the target task.
- Target > Supporting: Use STILTs. The supporting task serves as a beneficial "warm start" initialization, while the larger target dataset is sufficient for effective final fine-tuning.
This heuristic correctly identified the superior method in 92.5% of statistically significant cases. Controlled experiments further revealed a "crossover point": as the supporting task grows larger relative to the target, the performance advantage shifts decisively from STILTs to MTL.
Empirical Results
The study demonstrates that targeted pairwise transfer guided by the size heuristic significantly outperforms the common practice of training on all available tasks (MTLAll). While it is tempting to throw every dataset into one big multi-task hopper, the authors found that MTLAll was almost universally the worst-performing method, likely due to gradient interference between unrelated tasks.
Key findings include:
- Predictive Accuracy: The heuristic correctly predicted the better method in 49 out of 53 significant cases.
- Performance Gains: Implementing the size heuristic resulted in an average GLUE score of 78.3, a substantial improvement over the 73.3 achieved by
MTLAll. - Robustness: These patterns held across different model architectures, suggesting a fundamental characteristic of how encoders learn from multiple data sources.
Limitations and Future Work
The heuristic encountered an exception with the MRPC dataset, indicating that task similarity or specific linguistic features can occasionally override the influence of dataset size. Additionally, while the empirical evidence is compelling, the underlying theoretical mechanisms—such as the interplay between catastrophic forgetting in sequential training and gradient interference in joint training—remain an area for future research. Extending this heuristic to optimize clusters of three or more tasks also presents a promising direction for further study.
Summary updated