In machine learning, we often hope that learning many related tasks will make it easier to learn a new one. This is the core promise of few-shot learning, meta-learning, and multitask learning. While practitioners have long used "shared backbones" to achieve this, a recent wave of theoretical research has pinpointed the exact mathematical engine that makes this possible: task diversity.
The intuition is simple: if you want to learn a general-purpose representation of the world, you cannot look at it from only one angle. You need a collection of tasks that "triangulate" the underlying structure.
The Shared Subspace Hypothesis
At the heart of this research is a specific model of how tasks relate to one another. In papers like Few-Shot Learning via Learning the Representation, Provably, Simon S. Du and colleagues formalize the idea that while different tasks (like identifying a dog vs. identifying a cat) have different labels, they share a common feature extractor.
Mathematically, if $x$ is your input, a task $t$ is often modeled as $y = w_t^T B x$. Here, $B$ is a shared low-dimensional representation (the "backbone") and $w_t$ is a task-specific weight vector (the "head"). The goal of multi-task learning is to use many tasks to recover $B$. Once $B$ is known, learning a new task only requires finding a new $w_t$, which takes very little data.
Why Diversity is the Secret Sauce
If all your training tasks are nearly identical, you might learn the features necessary for those specific tasks, but you won't learn the full, robust representation $B$. You will have "blind spots."
Chi Jin, Nilesh Tripuraneni, and Michael I. Jordan demonstrate in Provable Meta-Learning of Linear Representations that the ability to recover $B$ depends on the condition number of the task diversity matrix.
Think of it this way: if your tasks are vectors in a space, you need them to "span" that space. If they all point in the same direction, you only see one dimension of the representation. High task diversity ensures that the training tasks explore all directions of the feature space, preventing the model from becoming "blind" to features required for future, unseen tasks.
The $1/nt$ Benefit
The most striking result of this theory is the dramatic reduction in sample complexity. In On the Theory of Transfer Learning: The Importance of Task Diversity, the authors prove that with sufficient diversity, the error in the learned representation $B$ vanishes as the total number of samples across all tasks ($n$ samples per task $\times$ $t$ tasks) increases.
This means that even if you have very little data for any single task (small $n$), you can still learn a near-perfect representation if you have enough diverse tasks (large $t$). This provides a rigorous explanation for why "pooling" data across tasks works: it’s not just about having more data; it’s about having data that covers more of the underlying manifold.
From Theory to Algorithms
This theoretical insight isn't just abstract; it explains why certain practical "tricks" work and suggests new ones.
- Curriculum Learning: In On the Statistical Benefits of Curriculum Learning, Ziping Xu and Ambuj Tewari show that when tasks share a structure, an effective curriculum should prioritize diversity over similarity. Selecting tasks that explore new directions in the shared space is more valuable than simply picking tasks that are "easy" or similar to what the model already knows.
- Regularization: Quentin Bouniot and colleagues, in Improving Few-Shot Learning Through Multi-task Representation Learning Theory, introduced a spectral-based regularization term. By explicitly forcing the task-specific predictors to be diverse (minimizing the condition number), they were able to boost the performance of standard meta-learning algorithms like MAML.
- Structural Robustness: Interestingly, multitasking also makes models more robust to physical damage or pruning. Giannis Daras and Alex Dimakis show in Multitasking Models are Robust to Structural Failure that this robustness is contingent on task diversity. If tasks are near-orthogonal, the model develops an implicit $L_2$ regularization that protects it from structural failures.
The Limits of Representation Transfer
The theory also warns us when multi-task learning will fail. If the target task requires features that were never "activated" by the source tasks, the learned representation will be useless. As noted in Guarantees for Nonlinear Representation Learning by Thomas TCK Zhang and others, there must be a "distributional coverage" where the aggregate of source tasks covers the input space of the target task.
Without this coverage—or if the early tasks in a sequence are redundant—standard algorithms fail to identify the full representation, leading to poor performance on future tasks that require those missing dimensions. Success in few-shot learning is thus not just a matter of "learning to learn," but a matter of having seen enough of the world's variety to know which features matter.