Abstract
Multitask Learning (MTL) is a cornerstone of modern deep learning, yet its empirical success is often unpredictable, leading to either significant gains or "negative transfer." This paper adopts a "Keplerian" approach to demystify MTL by treating model performance as a physical system. Through a combination of controlled task simulation and symbolic regression, the authors derive human-interpretable mathematical formulas that quantify how sample size, task relatedness, and the number of tasks influence $F_1$ scores. The findings not only rediscover theoretical convergence rates empirically but also provide actionable insights into the trade-offs between data quantity and task quality.
Introduction
In natural language processing (NLP), training models on multiple related tasks is a common strategy to bolster generalization, particularly when target task data is limited. However, the field lacks a precise understanding of when MTL will succeed. Existing theoretical frameworks often rely on assumptions—such as perfectly balanced datasets or known task relatedness—that are rarely met in practice. This research seeks to bridge the gap between abstract theory and empirical reality by discovering the "laws of motion" for MTL performance.
Methodology
The authors propose a robust three-step empirical methodology designed to isolate and quantify the drivers of MTL performance:
- Task Simulation: A custom simulator generates synthetic sequence labeling tasks based on the CoNLL-2003 NER dataset. By manipulating transition matrices, the researchers can precisely control variables such as the number of tasks ($T$), samples per task ($n$), and task relatedness.
- Large-Scale Empirical Training: Over 10,000 Bi-LSTM models were trained across these simulated environments. The resulting $F_1$ scores were recorded alongside their corresponding input parameters to create a comprehensive meta-dataset.
- Symbolic Regression: Using Genetic Programming (via the
gplearnlibrary), the authors searched for the simplest mathematical expressions that accurately fit the observed performance data.
The underlying model architecture consists of a standard Bi-LSTM with shared layers and task-specific heads, ensuring that the discovered formulas reflect common MTL practices.
Experimental Framework
The study utilizes the CoNLL-2003 Named Entity Recognition (NER) dataset as its real-world foundation. The simulator generates thousands of artificial tasks related to these NER labels, allowing for the creation of a "Meta-Dataset." In this dataset, each observation represents a specific configuration (e.g., $n=2000, T=5, AMI=0.6$) and its resulting average $F_1$ score. Task relatedness is quantified using Adjusted Mutual Information (AMI), providing a concrete metric for a traditionally vague concept.
Results and Discussion
The symbolic regression process yielded several simple, insightful formulas. For balanced datasets, the regressor identified that performance scales predictably with the square root of the sample size ($F_1 \approx \sqrt{n} \cdot 0.69$) and task relatedness ($F_1 \approx \sqrt{AMI} / 0.015$).
Key Insights
- Empirical Validation of Theory: The symbolic regressor "rediscovered" theoretical convergence rates, finding that performance scales with $\sqrt{n}$ and $\sqrt{T}$. This aligns with formal mathematical proofs, providing strong empirical support for existing theory.
- The Criticality of Relatedness: The results suggest that task relatedness ($AMI$) is as vital as data volume. In low-resource settings, a single highly related task can be more beneficial than multiple weakly related ones.
- Predictability in Unbalanced Settings: In realistic scenarios where the target task is data-poor but auxiliary tasks are data-rich, the benefit of MTL is nearly guaranteed, provided there is at least moderate task relatedness.
Conclusion
This work demonstrates the power of a "Keplerian" approach to machine learning. By observing data and fitting simple functions, the authors have provided a clearer picture of the dynamics governing Multitask Learning. While the current formulas are specific to sequence labeling and Bi-LSTM architectures, the methodology offers a blueprint for future research into Transformers and other domains. The complexity of the "all-in-one" formula suggests that while individual factors follow simple laws, their interactions remain a rich area for further investigation.
Summary updated