Training one network to predict several things about an image, such as per-pixel class labels, depth, and surface orientation, is cheaper and often more accurate than training one network per task. The catch is that real datasets rarely label every task on every image. A given image might have a depth map but no segmentation. The standard fix is to skip the loss for any missing label. That leaves the network unsupervised on exactly the tasks it most needs help with.
This note proposes borrowing a tool from multi-view clustering, the field that groups data points by combining several independent descriptions of each one. We treat each task's current prediction map as one "view" of the image's pixels. For any task that is unlabeled on an image, we cluster the pixels into regions using only the other tasks' views, and then require the unlabeled task's prediction to be coherent inside those regions. The supervision is about structure ("these pixels belong together"), not about values ("this pixel has depth 2.3 m"). Because the regions for task $t$ never use task $t$'s own prediction, a task cannot confirm its own mistakes. This is the failure that makes ordinary pseudo-labeling drift.
Core Claim
Agreement among the other tasks is a cheap, non-generative source of supervision for a missing task. On standard partially labeled dense benchmarks, it should close a substantial part of the gap between loss masking and heavy cross-task models. It should also add to, rather than duplicate, the gains of latent-space regularizers. The claim has two parts. The first is that leave-one-task-out consensus regions carry information that color-based superpixels or single-feature clustering do not, for example depth discontinuities between surfaces of the same color. The second is that choosing the region granularity per image with a reliability index, rather than fixing it, is what prevents drift.
Where This Sits in Prior Work
Four lines of work on partially labeled dense multi-task learning motivate the gap, and they share one trait: they model cross-task relations with learned parametric machinery.
- Learning Multiple Dense Prediction Tasks from Partially Annotated Data maps pairs of task predictions into a shared latent space through a mapping network conditioned by FiLM, a layer that rescales features depending on which task pair is being mapped.
- Joint-Task Regularization for Partially Labeled Multi-Task Learning (JTR) replaces those pairwise maps with a single auto-encoder over all tasks.
- DiffusionMTL refines imperfect predictions with a diffusion model, a generator that iteratively denoises its input, at the cost of iterative inference.
- HiTTs discovers missing labels with learnable task tokens, arguing explicitly that pseudo-labeling suffers from confirmation bias.
Our proposal adds no network at inference and no generative decoder. It is a training-time loss computed from a clustering of the predictions the model already makes.
The closest prior work is Self-Supervised Learning in Multi-Task Graphs through Iterative Consensus Shift (CShift). CShift builds per-pixel pseudo-labels by ensembling expert predictions weighted by how much they agree. It operates on values, pixel by pixel, in a domain-adaptation setting. We instead produce region partitions and supervise only within-region coherence, so the missing task's actual values remain free. Robust Learning Through Cross-Task Consistency enforces consistency along chained task-to-task mappings. That approach requires those mappings to exist and be trained, which ours does not.
From multi-view clustering we reuse three techniques:
- Mutual supervision. Multi-View Utility-Based Clustering shows that one view's partition can serve as auxiliary supervision for another view.
- Granularity choice by reliability. Multi-View Clustering with Granularity-Aware Pseudo Supervision (GAPS) argues that confirmation bias in pseudo-supervision comes from fixed granularity rather than from discrete assignments themselves. GAPS scores how reliable a view is with a Separation-Compactness Index (SCI).
- Anchor bipartite graphs. Automatic and Aligned Anchor Learning Strategy for Multi-View Clustering (3AMVC) connects all $N$ points to a small set of $M$ representative anchors, so clustering costs $O(NM)$ instead of $O(N^2)$.
GAPS directly contradicts HiTTs' premise that discrete pseudo-supervision must be abandoned. Our experiments are designed to settle that disagreement in the dense multi-task setting. GAPS also leaves the choice of its granularity parameter $\lambda$ open. We make that choice automatic per image.
The pairing "multi-task + multi-view clustering" has been explored before, but in the opposite direction. Learn Multi-task Anchor (JVILG) frames incomplete multi-view clustering as two jointly optimized tasks, imputation and label generation. It uses multi-task ideas to help clustering. We use clustering to help multi-task learning with missing labels, which none of those papers address.
Research Program
Views. Take an image with $K$ tasks, of which some subset $U$ is unlabeled. At feature resolution (roughly 1/4 of input), each task head $t$ produces a map $f_t$. The views are the detached maps $f_v$ for $v \neq t$, plus the backbone feature map as a task-agnostic view. "Detached" means gradients do not flow back through them.
Regions for each missing task. For each $t \in U$ we build an anchor bipartite graph per view with $M = 64$ anchors initialized on a grid. We fuse the per-view graphs with weights from each view's SCI and merge anchors hierarchically into candidate partitions at several granularities. We keep the partition $P^{(-t)} = \{R_1, \dots, R_C\}$ whose consensus SCI is highest. Per image this costs $O(NM)$, small next to a forward pass.
Coherence loss. For a categorical task, let $\bar{p}_c$ be the region-mean predicted class distribution in region $R_c$ and $w_c$ a reliability weight. The loss pulls each pixel's distribution toward its region mean:
$$ \mathcal{L}_t = \sum_{c=1}^{C} w_c \sum_{i \in R_c} \mathrm{KL}\big(\bar{p}_c \,\|\, p_t(i)\big) $$
Here $p_t(i)$ is the predicted class distribution at pixel $i$, and $w_c$ is the region's mean cross-view agreement.
For continuous tasks such as depth and normals, constant values inside a region would wrongly flatten slanted floors. We therefore apply the same dispersion penalty to the spatial gradient of $f_t$ rather than to $f_t$ itself. This permits planar slopes and penalizes kinks inside a region.
The total objective is the supervised loss on the labels that exist, plus $\beta \sum_{t \in U} \mathcal{L}_t$. Partitions are recomputed every iteration from detached predictions. Because the regions are built from other tasks, the trivial solution of a constant prediction does not satisfy the losses of the labeled tasks, and the regions cannot follow it.
Analyses. First, we measure partition quality directly, in the style of multi-view clustering evaluation. We compute NMI (normalized mutual information, a score of how well two partitions agree) between $P^{(-t)}$ and ground-truth segments, and the boundary recall of $P^{(-t)}$ against true depth discontinuities. This tests whether combining task views yields better regions than any single view, the multi-view analogue of Feature concatenation multi-view subspace clustering's finding that structured consensus beats naive concatenation. Second, we track drift by comparing partition quality and task error across training under fixed-granularity and adaptive-granularity variants.
Baselines
Loss masking. The naive reference is a shared encoder with per-task heads, trained only on the labels that exist. It is the starting point all prior methods report against. Learning Multiple Dense Prediction Tasks from Partially Annotated Data reports this baseline at 69.50 mIoU on Cityscapes one-label. A single-task-per-network reference is also trained, since gains in this literature are reported as the relative multi-task gain $\Delta_{MTL}$ over single-task models.
JTR. The strongest feed-forward competitor is JTR, because like our method it is a training-time regularizer with no inference overhead. On NYUD-v2 random-labels it reports +21.92% relative improvement over the supervised baseline, against +19.66% for the earlier MTPSL method.
HiTTs and DiffusionMTL. We report HiTTs and DiffusionMTL as stronger but heavier reference points. HiTTs reports +16.91% $\Delta_{MTL}$ on NYUD-v2. DiffusionMTL reports a $\Delta_m$ of +3.65% on PASCAL-Context one-label.
Fairness and the win condition. These published numbers come from different settings and protocols, so we will rerun JTR and HiTTs from released code where available. All methods share the same backbone, label splits, schedule, and a matched hyperparameter search of 12 trials each. We count beating JTR as two outcomes. Either JTR-plus-ours exceeds JTR alone on $\Delta_{MTL}$ beyond seed variance (three seeds), or ours alone matches JTR with lower training memory.
Ablations of the key ingredients. These isolate what matters:
- Regions built from all views including task $t$, removing the leave-one-out property.
- Regions from fixed-granularity clustering.
- Regions from backbone features only, removing the multi-task views.
- SLIC color superpixels, a classical clustering of pixels by color and position, substituted for consensus regions.
Why not just…?
Why not just mask the missing losses? This is the default, and the evidence against it is consistent. DiffusionMTL reports that masking leaves the shared encoder with noisy, inconsistent gradients across tasks. The Cityscapes gap from 69.50 to 74.90 mIoU in Learning Multiple Dense Prediction Tasks from Partially Annotated Data shows how much cross-task signal masking leaves unused. Our masking baseline measures directly how much of that gap the coherence loss recovers.
Why not just use color superpixels or any off-the-shelf segmentation as the region prior? This is the real threat, because region-coherence priors are old and might explain most of the gain. Our bet is that other tasks' predictions mark boundaries that color misses, and the reverse. The SLIC and backbone-only ablations settle this. If SLIC regions match leave-one-task-out regions on both task metrics and boundary recall at depth discontinuities, the multi-view contribution is not real. We would then report the result as a cheap region prior rather than a cross-task method.
Why not just self-train on confident pseudo-labels for the missing task? Value pseudo-labels from the task's own head feed its errors back to it. HiTTs argues this causes drift, and GAPS reports +22.29% ACC over pseudo-label-driven clustering methods when granularity is adapted. We include a confidence-thresholded self-training baseline. The "include task $t$" ablation tests the specific claim that excluding a task's own view is what blocks confirmation bias.
Why not just ensemble the other tasks per pixel, as CShift does? CShift reduces depth L1 on Replica from 14.58 to 12.77, so per-pixel consensus clearly helps. However, it needs predictions that can be converted into the target task's values. We reimplement a CShift-style per-pixel ensemble on NYUD-v2. Doing so requires the cross-task mapping networks that our method avoids, and we compare against it at equal compute.
Proposed Experiments
Experiments run on NYUD-v2 (segmentation, depth, normals), Cityscapes (segmentation, depth), and PASCAL-Context (five tasks), under the one-label and random-label partial-annotation protocols used by JTR and HiTTs. We report mIoU, depth RMSE, mean normal angle error, and $\Delta_{MTL}$, along with training time, peak memory, and inference latency.
Three checks decide the outcome:
- Gain over masking and additivity with JTR. Our loss should recover a clear share of the masking-to-JTR gap on NYUD-v2 random-labels, and JTR-plus-ours should exceed JTR alone.
- Ablation ordering. Leave-one-out consensus should beat SLIC and backbone-only regions on depth-discontinuity boundary recall and on task metrics.
- Drift under fixed granularity. Fixed granularity should show the predicted pattern: partition NMI falls over training while the adaptive variant holds steady.
If the first check passes but the second fails, the honest conclusion is that region priors help partially labeled multi-task learning but cross-task views do not add to them. If check 3 fails, GAPS' account of confirmation bias does not transfer to dense multi-task learning, and HiTTs' position is strengthened.
References
- Learning Multiple Dense Prediction Tasks from Partially Annotated Data
- Joint-Task Regularization for Partially Labeled Multi-Task Learning
- DiffusionMTL: Learning Multi-Task Denoising Diffusion Model from Partially Annotated Data
- Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions
- Self-Supervised Learning in Multi-Task Graphs through Iterative Consensus Shift
- Robust Learning Through Cross-Task Consistency
- Multi-View Utility-Based Clustering: A Mutually Supervised Perspective
- Multi-View Clustering with Granularity-Aware Pseudo Supervision
- Automatic and Aligned Anchor Learning Strategy for Multi-View Clustering
- Learn Multi-task Anchor: Joint View Imputation and Label Generation for Incomplete Multi-view Clustering
- Feature concatenation multi-view subspace clustering