When we train a machine learning model on a source task and transfer its weights to a target task, we make a massive, often unexamined assumption: that the two tasks are fundamentally related. If they are, the model enjoys a "jump start" in performance and faster convergence. If they are not, the model suffers from negative transfer—a phenomenon where prior knowledge actively degrades performance, making it worse than learning from scratch.
Historically, machine learning practitioners have relied on human intuition to decide which tasks are related. We assume that classifying cats is related to classifying dogs, or that sentiment analysis is related to question answering. However, as deep learning scales to multi-task networks, lifelong learning agents, and foundation models, human intuition fails. We need a formal, mathematical way to answer a deceptively simple question: How far apart are two learning tasks?
The Phase Transition of Knowledge Transfer
To understand why formal metrics are necessary, we must look at how transfer learning behaves under the hood. It is tempting to think of task similarity as a smooth, linear gradient where a little bit of similarity yields a little bit of transfer benefit. However, theoretical work reveals a more volatile reality.
By analyzing high-dimensional learning systems, Dhifallah et al. (2021) proved that transfer learning undergoes a sharp phase transition. Transferring knowledge only becomes beneficial once the mathematical similarity between the source and target tasks crosses a specific, hard threshold. Below this threshold, the source task's representations act as geometric noise, actively harming the target model.
This phenomenon is a core bottleneck across machine learning paradigms:
- Meta-Learning: A lack of formal metrics makes it difficult to predict whether a model will generalize to a novel optimization landscape or result in suboptimal knowledge reuse (Yu (2025); Vanschoren (2018)).
- Lifelong Learning: Where an agent encounters a sequence of environments, the absence of task-distance metrics prevents us from rigorously evaluating why an agent forgets certain skills while retaining others (Nekoei et al. (2021)).
As Tadesse et al. (2020) demonstrated mathematically, favorable transfer resources are rare in the wild. Without a way to measure task affinity, we are searching for a needle in a haystack in the dark.
The Ideal Metric: Algorithmic Complexity
What does a perfect task metric look like? Ideally, it should measure the intrinsic informational overlap between two problems, independent of the specific neural network architecture used to solve them.
An early, foundational answer came from Mahmud et al. (2007), who proposed using conditional Kolmogorov complexity as a formal metric for task relatedness. In algorithmic information theory, the Kolmogorov complexity of an object is the length of the shortest computer program that can generate it. Extending this to tasks, the conditional Kolmogorov complexity measures how much additional computational information is required to solve a target task given that you already possess the optimal program for a source task.
If two tasks share a deep algorithmic structure, the conditional complexity is low, meaning the transition from one to the other requires very little new code. While this framework provides an elegant proof of universal optimality for sequential transfer, it suffers from a fatal flaw: Kolmogorov complexity is fundamentally uncomputable. It sets a theoretical gold standard, but practitioners cannot use it to calculate the distance between two modern image datasets.
Data-Driven and Representation-Based Distances
To bridge the gap between theory and practice, researchers have developed empirical proxies that capture how a model's internal representations respond to different tasks.
One approach is to build an empirical "map" of transferability by running brute-force experiments. In their seminal work on Taskonomy, Zamir et al. (2018) bypassed abstract mathematics by training models on dozens of distinct visual tasks (such as depth estimation, edge detection, and classification) and measuring the actual performance gain when transferring features between every possible pair. This created a data-driven affinity matrix, revealing a hidden taxonomic structure of visual features.
While powerful, brute-force mapping is computationally exorbitant. To find a cheaper metric, Cao et al. (2023) introduced a method called AutoTransfer, which computes a "task embedding" based on Fisher Information. Instead of training a network to convergence on every task, they measure how the gradients of a probe model behave when exposed to a new dataset. The Fisher Information matrix acts as a statistical signature of what the network needs to learn, allowing researchers to instantly calculate task similarity by measuring the distance between these matrices.
Similarly, in lifelong learning, Aljundi et al. (2017) utilized lightweight gating autoencoders to capture the feature distribution of each incoming task. By measuring how well an autoencoder trained on a previous task reconstructs the data of a new task, the system obtains a real-time measure of feature distribution overlap, allowing it to autonomously route data to the most relevant expert model.
[Task A Data] ---> [Autoencoder A] ---> Low Reconstruction Error ===> High Similarity
[Task B Data] ---> [Autoencoder A] ---> High Reconstruction Error ===> Low Similarity
Bridging Heterogeneous Spaces via Attributes and Kernels
A major hurdle arises when tasks do not share the same input distributions, feature spaces, or label sets. For instance, how do you measure the similarity between an unsupervised pre-training graph task and a supervised downstream node classification task? As Huang et al. (2024) noted, they lack a common mathematical denominator.
To resolve this, researchers have turned to intermediate abstractions. One solution is Task Attribute Distance (TAD), introduced in a 2023 study on few-shot learning. Instead of comparing raw data or model weights, TAD projects tasks into a shared semantic attribute space. By calculating the distance between these high-level attributes, the metric provides a model-independent way to predict generalization error before a single parameter is updated.
For scenarios where tasks are completely heterogeneous, Low (2021) proposed a Gaussian Process Meta-Learning framework that employs a specialized task kernel. This kernel learns to map disparate task spaces onto a set of shared, representative "inducing tasks," enabling the model to calculate a functional distance between problems that superficially look entirely different.
This structural alignment is critical for modern foundation models. For example, Letey et al. (2026) derived an analytical alignment metric that measures how closely the structural geometry of pre-training tasks matches test-time prompts, directly predicting the success of in-context learning.
Tasks as a Continuous Flow
Most frameworks treat tasks as discrete, isolated points in a metric space. However, a recent paradigm shift reimagines task relationships as a continuous trajectory.
Ye et al. (2023) introduced NORMAL, a multi-task learning framework based on Neural Ordinary Differential Equations (Neural ODEs). Instead of assigning separate heads or hard boundaries to different tasks, the model treats multi-task feature extraction as a continuous dynamic flow. Each task is mapped to a specific "time point" along this ODE trajectory.
Continuous ODE Flow: t=0 -------------------> t=1 -------------------> t=2
| | |
[Task 1 Position] [Task 2 Position] [Task 3 Position]
The distance between these temporal positions provides a natural, geometric measure of task similarity. If two tasks are highly related, their optimal time points collapse close together, allowing them to share the vast majority of the continuous representation space. If they diverge, their positions drift apart along the flow. This continuous formulation provides an elegant solution to parameter competition, proving that task similarity doesn't have to be a static lookup table—it can be a dynamic coordinate in a shared learning space.
Key Foundational Work
Intelligent bias of network structures in the hierarchical BOA (2009) — contributed 2 concept elements to this direction.
Transfer Learning using Kolmogorov Complexity: Basic Theory and Empirical Evaluations (2007) — contributed 1 concept elements to this direction.