Multi-Task Learning (MTL) operates on a powerful premise: by training a single model to perform multiple related tasks simultaneously, the model learns a shared representation that is more robust and generalizes better than any single-task counterpart.
However, traditional MTL frameworks rely on a fragile assumption: the availability of fully paired datasets where every single training sample contains ground-truth labels for every target task. In the real world, this assumption quickly falls apart. Labeling dense predictions (like segmentation masks and depth maps) or specialized domain data is prohibitively expensive. Instead of a perfect, fully annotated matrix, researchers are forced to work with partially annotated or disjoint datasets—where Dataset A contains labels only for Task A, and Dataset B contains labels only for Task B.
When concurrent ground-truth labels are missing, standard multi-task architectures suffer from a supervision gap, leading to noisy gradients, spatial incoherence, or negative transfer. To unlock the true potential of multi-task models without the bottleneck of exhaustive annotation, a major research direction focuses on learning cross-task correlations from incomplete supervision.
The Core Challenge: The Missing Label Matrix
To understand why incomplete supervision breaks standard multi-task learning, imagine training a network to predict both semantic segmentation and depth from images. In a standard setup, the loss function computes gradients for both tasks on every image, forcing the shared encoder to find a feature space that satisfies both objectives simultaneously.
As noted by Deblina Bhattacharjee et al. in MuIT: An End-to-End Multitask Learning Transformer, this requirement creates a massive scalability bottleneck. When you transition to partially annotated data, a naive approach is to simply mask out the loss for the missing labels on a given sample. However, as Hanrong Ye et al. demonstrate in DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data, this causes the model to struggle to learn consistent representations across different output domains. Without explicit cross-task supervision on the same image, the model cannot easily exploit the complementary nature of the tasks, often causing it to underperform relative to simple single-task baselines.
Furthermore, real-world data often arrives as completely disjoint sets. As Li Niu et al. observe in Beyond without Forgetting: Multi-Task Learning for Classification with Disjoint Datasets, this forces models to learn from incomplete supervision across entirely different data distributions, compounding the challenge of catastrophic forgetting with domain shift.
Mechanism 1: Latent Space Alignment and Task Tokens
If we cannot correlate tasks explicitly through paired ground-truth labels, how can the model learn how tasks relate? One prominent solution is to project predictions or intermediate features into a shared latent space where cross-task relationships can be regularized implicitly.
Instead of computing a loss directly against missing labels, Kento Nishi et al. introduce Joint-Task Regularization (JTR) in Joint-Task Regularization for Partially Labeled Multi-Task Learning. JTR maps all task predictions into a single, shared latent space. By optimizing the relationships between tasks within this unified space, the model learns cross-task correlations with linear efficiency, bypassing the need for concurrent ground truth labels.
[Input Image] ---> [Shared Encoder] ---> [Task Heads] ---> [Individual Predictions]
|
v
[Shared Latent Space Alignment]
(Learns cross-task correlations)
Taking this concept further, Hanrong Ye et al. developed Hierarchical Task Tokens (HiTTs) in Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions. HiTTs utilize learnable, compact tokens to discover missing supervision signals in both the feature and prediction spaces. These tokens act as contextual bridges, capturing task dependencies across the entire dataset even when individual images lack labels for specific objectives.
Similarly, Dimitrios Kollias et al. show in Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond that leveraging known statistical relationships between tasks allows models to be trained effectively on non-overlapping datasets, preventing the "negative transfer" that typically occurs when gradients are missing.
Mechanism 2: Noise-Aware Pseudo-Labeling and Self-Training
Another intuitive way to handle missing annotations is to transform the problem into a semi-supervised or self-training regime: let the model generate its own supervision for the missing tasks.
In the context of disjoint classification datasets, Li Niu et al. propose MTL with Selective Augmentation (MTL-SA). Instead of ignoring the unlabeled portions of disjoint datasets, MTL-SA actively uses them by generating pseudo-labels. To prevent the model from reinforcing its own mistakes, they employ a noise-aware pseudo-labeling strategy that filters out low-confidence or highly ambiguous predictions.
This strategy translates effectively to natural language processing. For instance, Timothée Bernard combines self-training and reinforcement learning with a custom reward function to train joint part-of-speech taggers and parsers on disjoint corpora.
When gold-standard labels are scarce for a primary task, auxiliary tasks can be used to anchor the self-training process. Isabelle Augenstein et al. demonstrate this in Transductive Auxiliary Task Self-Training for Neural Multi-Task Models. By using a single-task model to generate "silver" labels for an auxiliary task directly on the target test set, the multi-task model can exploit cross-task benefits even when the primary objective has minimal training data.
Mechanism 3: Generative Rectification and Logical Consistency
When a model is trained on partially annotated data, its raw predictions for the unannotated tasks are often plagued by visual artifacts, noise, or structural inconsistencies. A third major mechanism addresses this by enforcing generative priors or logical constraints over the output space.
In DiffusionMTL, Hanrong Ye et al. reframe multi-task learning with partial labels as a generative denoising problem. Instead of treating task outputs as independent predictions, the framework applies a denoising diffusion model to the initial, noisy task maps. By conditioning the diffusion process on the available tasks, the model effectively "rectifies" the missing or poorly predicted task outputs, ensuring spatial coherence and alignment across domains.
[Partial Labels] ---> [Initial Prediction] ---> [Diffusion Denoising Network] ---> [Rectified, Coherent Outputs]
^
|
[Multi-Task Conditioning]
Alternatively, knowledge about the physical or logical relationship between tasks can act as a powerful regularizer. In specialized domains like surgical video analysis, Yueming Jin et al. demonstrate in Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis that a specialized "correlation loss" can enforce logical consistency between tasks (such as tool detection and surgical phase recognition). By explicitly penalizing conflicting predictions, the model uses the logical structure of the problem to guide learning when one or more task labels are missing.
Summary
Multi-task learning under incomplete supervision shifts the focus from data collection to architectural and algorithmic cleverness. Whether through aligning tasks in latent spaces, selectively generating pseudo-labels, or applying generative denoising priors, these methods allow deep networks to map the underlying correlations of our world without requiring a perfect, all-encompassing dataset.