Multi-Task Learning from Partially Annotated Data

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2160 papers182 conceptsfrozen 2026-10-04

In an ideal world, every training image for a multi-task model would come with a complete set of labels: a segmentation mask, a depth map, surface normals, and object boundaries. In reality, such "fully annotated" datasets are rare and prohibitively expensive to produce. Most real-world data is fragmented—one dataset might provide depth labels from LiDAR, while another offers manually labeled semantic masks, but they rarely overlap on the same pixels.

Standard Multi-Task Learning (MTL) assumes that for every input, we have ground truth for every task. When labels are missing, the model loses the ability to learn the inherent correlations between tasks—like the fact that depth usually changes abruptly at object boundaries. Recent research has shifted toward Multi-Task Partially Supervised Learning (MTPSL), developing mechanisms to "hallucinate" missing supervision and maintain cross-task synergy despite incomplete data.

The Challenge of Disjoint Supervision

When a model is trained on partially annotated data, the most common approach is to simply mask the loss for missing tasks. However, this often leads to the "seesaw effect," where the model improves on one task at the expense of others. Without simultaneous labels, the network cannot easily learn how tasks constrain one another.

As noted by Weihong Li and Hakan Bilen in Learning Multiple Dense Prediction Tasks from Partially Annotated Data, this lack of direct supervision for missing tasks leads to poor generalization. The goal of this research direction is to bridge these gaps by treating missing labels not as "null" values, but as latent variables that can be inferred from the tasks that are present.

Bridging Gaps with Shared Task Spaces

One powerful intuition for solving this is to map different task predictions into a shared representation. If the model knows that Task A (segmentation) and Task B (depth) are geometrically related, it can use its prediction for Task A to "regularize" its prediction for Task B, even if the ground truth for Task B is missing.

Weihong Li and colleagues proposed a framework that maps predictions into a joint pairwise task-space. By enforcing consistency between these mapped predictions, the model effectively "hallucinates" the missing supervision. Similarly, Kento Nishi and Wanhua Li introduced Joint-Task Regularization (JTR) in Joint-Task Regularization for Partially Labeled Multi-Task Learning. JTR projects task predictions into a shared latent space to learn cross-task relationships with linear efficiency, allowing the model to scale to many tasks without the quadratic complexity of comparing every task pair.

Latent Discovery and Task Tokens

Rather than just enforcing consistency, some researchers use learnable components to "discover" the relationships between tasks. Hanrong Ye and Dan Xu introduced Hierarchical Task Tokens (HiTTs) in Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions.

HiTTs use compact, learnable tokens to capture task-specific information. These tokens interact with each other to discover missing supervision signals in both the feature and prediction spaces. This allows the model to bridge the "supervision gap" by leveraging the shared knowledge encoded in these tokens, outperforming standard models that rely on rigid, fully-labeled architectures.

Generative Rectification and Distillation

A more recent trend involves using generative frameworks to "clean up" the noisy predictions that result from partial supervision. DiffusionMTL reframes MTL as a denoising problem. By applying a diffusion process to initial task outputs, the model can "rectify" its predictions. The denoising process is conditioned on all available tasks, allowing the model to share information across domains even when specific labels are absent.

Another common strategy is teacher-student distillation. In domains like affective computing, where labels for facial expressions and "Action Units" are often disjoint, researchers like Didan Deng and Bertram E. Shi use a teacher model to generate "soft labels" for missing data. As shown in FAU, Facial Expressions, Valence and Arousal: A Multi-task Solution, students trained on a mix of ground truth and teacher-generated labels often surpass the performance of the teacher itself by learning more robust, integrated representations.

Asynchronous and Flexible Learning

In complex domains like healthcare, labels are not just missing; they are often asynchronous. A patient might have a diagnosis for one condition but not another. Muhao Xu and Zhenfeng Zhu addressed this with FlexCare, a framework that treats MTL as a series of asynchronous single-task updates. By using a task-guided "Mixture of Experts" (MoE) architecture, FlexCare allows the model to leverage a shared knowledge base while training on datasets where patients have varying subsets of clinical labels.

Why It Works: The Power of Implicit Constraints

The underlying intuition across all these methods is that tasks are not independent. The physical world imposes constraints: a sharp change in depth must correspond to an edge in the image; a specific facial muscle movement must correlate with a certain emotion.

By using shared latent spaces, task tokens, or generative denoising, these models turn partial supervision into a structured inference problem. Instead of seeing a missing label as a lack of information, these frameworks treat it as a puzzle piece that can be reconstructed by looking at the surrounding tasks. This allows AI to move beyond the "perfect data" requirement and learn from the messy, fragmented datasets that define the real world.

Papers in this version

Showing 60 of 160 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 107 papers
  2. Version 1 · 2026-10-04 · 160 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_multi_task_annotation_scarcity_d85d0446,
  title = {multi-task annotation scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-annotation-scarcity/rdr_d85d044630604ef482d3cb9ae783a935},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_task_learning_from_partially_annot_d85d0446_v1,
  title = {Multi-Task Learning from Partially Annotated Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-task-learning-from-partially-annotated-data/rdr_d85d044630604ef482d3cb9ae783a935/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}