Formal Metrics for Task Relatedness and Transferability

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2194 papers215 conceptsfrozen 2026-10-04

In modern machine learning, we rarely start from scratch. We fine-tune pre-trained models, share parameters across multiple objectives, and attempt to learn new skills without forgetting old ones. Yet, as noted by Joaquin Vanschoren in Meta-Learning: A Survey, the field lacks a universal, formal metric to define "relatedness" between tasks.

Without a rigorous way to measure task similarity, transfer learning remains a game of trial and error. We risk "negative transfer"—where irrelevant prior knowledge actually degrades performance—and "catastrophic forgetting," where new tasks overwrite old ones. To move beyond heuristics, researchers are developing a "metrology" for AI, seeking mathematical frameworks that can predict transferability before a single weight is updated.

The Information-Theoretic Ideal

The most fundamental way to think about task similarity is through the lens of algorithmic information theory. In Transfer Learning using Kolmogorov Complexity: Basic Theory and Empirical Evaluations, M. M. Hassan Mahmud and Sylvian R. Ray propose using conditional Kolmogorov complexity as a formal metric.

The intuition is simple: Task B is related to Task A if the shortest computer program required to solve Task B is significantly shorter when the program for Task A is already available. While Kolmogorov complexity is uncomputable in its pure form, this framing provides a "universally optimal" theoretical baseline. It suggests that transfer is almost always justified if we can find the shared "code" between problems.

Tasks as Points in Vector Space

If we cannot compute the exact complexity of a task, can we at least embed it into a manageable space? Task2Vec: Task Embedding for Meta-Learning treats an entire machine learning task as a single, fixed-dimensional vector.

By calculating the Fisher Information Matrix of a "probe" neural network trained on a specific dataset, researchers can create a signature of that task. This signature captures which parameters are most important for that specific objective. The distance between these vectors then serves as a proxy for task similarity, allowing practitioners to automatically select the best pre-trained model for a new problem without expensive fine-tuning.

The Geometry of Gradients and Kernels

In sequential and continual learning, the challenge isn't just finding a starting point, but avoiding interference. Thang Doan and colleagues explore this through the Neural Tangent Kernel (NTK) overlap matrix in A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix.

The intuition here is geometric: a neural network's "knowledge" of a task is represented by the subspace of gradients it uses to solve it. If the gradient subspaces of Task A and Task B are orthogonal, the network can learn Task B without overwriting the weights essential for Task A. Masashi Sugiyama and others use this to provide Generalisation Guarantees for Continual Learning, proving that a model's ability to generalize depends directly on how closely these task subspaces align.

Bridging the Semantic Gap

Sometimes, tasks don't share data or gradients, but they share concepts. Minyang Hu and co-authors address this in Task Attribute Distance for Few-Shot Learning. They introduce Task Attribute Distance (TAD), which uses a shared language of attributes (e.g., "has wings," "is metallic") to quantify relatedness.

This provides a theoretical bridge: if a new task requires attributes the model has already mastered in a different context, the "distance" is low, and the model will require fewer samples to adapt. This moves the metric from the "how" of the model (gradients) to the "what" of the domain (semantics).

Why Metrics Matter for Autonomy

The ultimate goal of this research is to build agents that can decide for themselves when to learn from others. As Felipe Leno da Silva and Anna Helena Reali Costa argue in their work on Multiagent Reinforcement Learning, autonomous agents need these metrics to move away from rigid teacher-student roles toward fluid, confidence-based knowledge sharing.

Whether it is Jing Yang using $\eta$-bracketing numbers to prove efficiency in Non-Markovian environments, or Cibu Mathew calling for a Metrology for Artificial Intelligence, the direction is clear: we are moving from an era of "blind" transfer to a science of task relationships. By quantifying the distance between problems, we can finally predict not just that a model will learn, but how its previous experiences will shape its future.

Papers in this version

Showing 60 of 194 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 106 papers
  2. Version 1 · 2026-10-04 · 194 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_task_similarity_quantification_4284320e,
  title = {task similarity quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-similarity-quantification/rdr_4284320e93c24a5697add28ea73eb661},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_formal_metrics_for_task_relatedness_and__4284320e_v1,
  title = {Formal Metrics for Task Relatedness and Transferability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/formal-metrics-for-task-relatedness-and-transferability/rdr_4284320e93c24a5697add28ea73eb661/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}