Order-Invariant Joint Likelihoods for Partially Labeled Multi-Task Learning

A multi-task network predicts several things about the same input, for example depth, surface orientation, and semantic labels for one image. Its separate output heads can easily disagree in ways that no real scene could produce. Probability theory gives a simple test of coherence. If the heads describe one joint distribution over all task outputs, then the probability of a full set of outputs must be the same whichever task you predict first and condition the others on. We propose to train multi-task models so that this holds. Each task head can optionally condition on the outputs of other tasks. The training loss then penalizes the spread, across randomly sampled task orderings, of the joint log-likelihood each ordering assigns. One shared conditional head per task replaces the quadratic number of pairwise translator networks that earlier consistency methods need. The constraint can also be checked on model-sampled values for tasks that were never labeled, which is exactly what partially labeled multi-task data needs.

The idea combines two existing pieces. Dual supervised learning showed that requiring $P(x)P(y|x)=P(y)P(x|y)$ improves both directions of an inverse task pair. Self-consistency in amortized Bayesian inference showed that a quantity that should be invariant can be enforced by penalizing its variance over arbitrary sampled values. Neither has been turned into a scalable, probabilistic consistency rule for $K$ tasks with heterogeneous outputs and missing labels. The most direct test bed is partially labeled dense prediction on NYU-v2 and Cityscapes, where the current best method relies on a deterministic autoencoder.

Core Claim

Let $y_S$ be the outputs of a subset $S$ of tasks for input $x$. Let $q(y_k \mid x, y_C)$ be the head for task $k$, given any conditioning subset $C$ of other tasks, with $C$ possibly empty. For an ordering $\pi$ of $S$, the joint the model implies along that ordering is

$$\log q_\pi(y_S \mid x) = \sum_{t=1}^{|S|} \log q\big(y_{\pi_t} \mid x, y_{\pi_1},\dots,y_{\pi_{t-1}}\big),$$

and a coherent model gives the same value for every $\pi$. The proposed regularizer is the empirical variance of this quantity over $M$ sampled orderings:

$$\mathcal{L}_{\text{order}}(x, y_S) = \mathrm{Var}_{\pi_1..\pi_M}\big[\log q_{\pi_m}(y_S \mid x)\big].$$

It is added to the usual likelihood terms with weight $\lambda$. Two claims follow.

The first is about the regularizer's mechanism. Every term is conditioned on $x$, so no external model of the input density is needed. That removes the known weak point of dual supervised learning, where errors in the auxiliary marginal estimators of $P(x)$ and $P(y)$ feed directly into the penalty.

The second is about missing labels. The identity holds for any values of $y$, not only observed ones, just as the Bayesian self-consistency loss holds for any parameter draw. So when sample $x$ has labels only for $S$, we draw the missing tasks from the model, $\tilde y_{\bar S} \sim q(\cdot \mid x, y_S)$, and apply $\mathcal{L}_{\text{order}}$ to the completed set. Unlabeled tasks then receive gradient signal tied to the labeled ones through an exact probabilistic identity rather than a learned distance.

One caveat shapes the whole design. A model whose conditional heads ignore their conditioning satisfies the constraint trivially. The constraint has teeth only when supervised likelihood forces the conditional heads to use other tasks' outputs. We therefore train conditional heads on ground-truth conditioning wherever labels co-occur, and we treat "do the conditionals collapse to the marginals?" as a measured failure mode, not an assumption.

Relation to Prior Work

The method comes from the duality line. Dual Supervised Learning enforces $P(x)P(y|x)=P(y)P(x|y)$ through a squared log-ratio penalty on an inverse pair. It reports, for example, CIFAR-10 error dropping from 6.43% to 5.40% when paired with PixelCNN++. Dual Supervised Learning for Natural Language Understanding and Generation needs a separate masked autoregressive density estimator just to supply $P(x)$ over slot sets. Both are limited to two invertible tasks and depend on marginal density estimates. We keep the log-space equality but move it to factorization orders among $K$ outputs of a shared input, so all terms are model conditionals. Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inference supplies the variance-over-draws form and the insight that consistency can be checked off the data, which yielded an 8-fold reduction in simulations on Two Moons. We reuse that form with orderings and sampled task completions in place of parameter draws.

Two works already apply probabilistic agreement inside multi-task systems, but in narrower form. Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement matches two chain-rule paths over a fixed triplet of sequences, and it needs complete triplets. Our proposal generalizes its two paths to random orderings over $K$ tasks and drops the completeness requirement. Joint Feature Encoding and Task Alignment Mechanism for Emotion-Cause Pair Extraction aligns a pair distribution with a product of marginals, which hard-codes conditional independence. Our conditionals model the dependence instead of assuming it away.

The multi-task consistency line motivates the problem and sets the bar. Robust Learning Through Cross-Task Consistency penalizes disagreement between direct and composite prediction paths. It uses deterministic pixel losses and $O(K^2)$ transfer networks, and its "consistency energy" detects out-of-distribution inputs with ROC-AUC 0.95. Joint-Task Regularization for Partially Labeled Multi-Task Learning gets to $O(K)$ by encoding the stacked predictions with one autoencoder. It is deterministic and models no likelihood. Our cost is $O(MK)$ evaluations of light heads on a backbone computed once, and the consistency signal is a likelihood identity.

Legal Judgment Prediction based on Knowledge-enhanced Multi-Task and Multi-Label Text Classification enforces article–charge coherence through a hand-built compatibility matrix. We test whether learned conditionals plus the order constraint recover that coherence without the dictionary.

Optimization-level methods are complementary rather than competing. Dual-balancing for multi-task learning and RotoGrad rescale or rotate gradients but treat heads as independent. Impartial Multi-task Representation Learning via Variance-invariant Probabilistic Decoding equalizes per-task latent variance without any cross-task joint. Deep Multivariate Models with Parametric Conditionals is the closest in spirit, since it also learns full conditionals over heterogeneous variables. It obtains coherence through MCMC with no control over mixing, and its CelebA-HQ model took about four days on an H100. We need no sampling chain at test time, because the unconditioned head $q(y_k\mid x)$ is the deployed predictor.

Research Program

The architecture is a standard shared encoder with one decoder per task. Each decoder also accepts a stack of other tasks' outputs, with a binary mask indicating which tasks are present and empty slots when $C$ is empty. Conditioning is injected through feature-wise modulation so the added cost is small.

Heads output proper per-pixel densities. Segmentation uses a categorical distribution. Depth uses a heteroscedastic Laplace distribution. Surface normals use a von Mises–Fisher distribution. For classification-style benchmarks, heads are categorical or Bernoulli. Per-pixel factorization within a task given its conditioning keeps every log-likelihood exact and cheap.

Training draws one random conditioning subset per head per step for the supervised conditional likelihood. It then draws $M$ orderings (default $M=4$) for $\mathcal{L}_{\text{order}}$. Missing tasks are completed by sampling from the model, with stop-gradient through the samples themselves. The log-variance is normalized by the number of pixels so that $\lambda$ transfers across resolutions.

The analysis has three parts. The first tracks an "information use" diagnostic: the gap between conditional and marginal negative log-likelihood on validation data. This detects the trivial solution in which conditionals stop using their inputs. The second sweeps $\lambda$ and $M$ to chart the trade-off between coherence and accuracy. The third evaluates the per-sample ordering variance at test time as an unsupervised error and out-of-distribution signal, the probabilistic counterpart of consistency energy.

Baselines

The naive reference point is equal-weight hard-parameter-sharing multi-task training on observed labels only, plus single-task models. Every method shares the same backbone, input resolution, epochs, and partial-label masks from the JTR protocol.

The strongest existing competitor on the main question is Joint-Task Regularization. It targets exactly partially labeled dense multi-task learning and reports a +21.92% relative improvement over the supervised baseline on NYU-v2 in the random-labels setting, ahead of MTPSL at +19.66%. We will rerun it from its released configuration on our backbone with the same hyperparameter search budget as ours, for example an equal number of trials over loss weights. Beating it means a higher relative multi-task gain averaged over three seeds on NYU-v2 and Cityscapes. Cityscapes has no reported JTR number on Lacuna, so the comparison there rests on our reruns. We will also report wall-clock training time, since JTR's advantage over MTPSL on Taskonomy was partly speed: 105 versus 223 hours.

For coherence-focused comparisons we add a pairwise-path consistency baseline in the style of Cross-Task Consistency on the three-task NYU-v2 graph, where $K^2$ remains affordable. For gradient-level competition we add DB-MTL and RotoGrad. For the latter, the one Lacuna number is a median normal angle of 20.80 versus 26.16 for vanilla on NYUv2 under full labels. We also test our loss combined with DB-MTL, because these methods address a different problem.

On CAIL2018 the reference is K-LJP at 61.41% Macro-F1 on multi-label charge prediction and a 97.93% alignment ratio. The claim is not that we beat it with a dictionary-free method. The claim is that we close most of the gap to it while using no legal mapping table.

Two ablations isolate the key ingredient. In the first, conditional heads are trained by supervised likelihood with $\lambda=0$. This tests whether gains come merely from the auxiliary conditional tasks. In the second, the order-variance is replaced by an L1 agreement between point predictions from different orderings. This tests whether the probabilistic form matters. A third ablation turns off sampled completion of missing tasks, which tests the partial-label mechanism specifically.

Why not just…?

Why not just use JTR's autoencoder, which is already linear in the number of tasks? It is a real threat on accuracy. Its distance-in-latent-space loss may capture most of the cross-task structure that matters for NYU-v2. What it cannot do is assign likelihoods, express per-sample uncertainty, or verify coherence. So if we only match it on accuracy, the contribution narrows to calibration and to the test-time consistency signal. The head-to-head rerun and the negative log-likelihood and calibration metrics settle this. We will report the outcome either way.

Why not just add conditional heads as auxiliary tasks and skip the constraint? Auxiliary prediction of one task from another may by itself regularize the shared encoder, and that would explain any gains without the duality idea. The $\lambda=0$ ablation answers this directly. If it matches the full method, the order constraint is not pulling its weight and we will say so.

Why not just pseudo-label the missing tasks with the model's own predictions? Self-training is the obvious answer to partial labels, and our sampled completion resembles it. The difference is that pseudo-labels supervise each missing task in isolation. Our completions are scored jointly with the labeled tasks, so a completion inconsistent with the observed depth is penalized. We include a confidence-thresholded pseudo-labeling baseline at matched compute. The sampled-completion ablation shows whether joint scoring adds anything beyond it.

Why not just balance gradients with DB-MTL or RotoGrad? These methods fix tasks fighting over the shared encoder. They say nothing about whether the outputs are mutually coherent, so they are not substitutes. The combined run tests whether the two effects add up. Measuring coherence separately from accuracy, through ordering variance and alignment ratio, shows that they target different failures.

Proposed Experiments and Decisive Checks

The main experiment is partially labeled NYU-v2 (segmentation, depth, normals) and Cityscapes, under JTR's random-labels and one-label-per-image protocols. We report per-task metrics, the relative multi-task gain, validation negative log-likelihood and expected calibration error, training time, and memory.

The scalability test is CelebA with 40 binary attributes. Here pairwise methods are infeasible and Lacuna reports no numeric RotoGrad F1, so the comparison rests on our reruns. We measure average F1 and the probability the model assigns to attribute combinations that never occur in training.

The heterogeneous-output test is CAIL2018 articles and charges. It measures Macro-F1 and the alignment ratio computed with the legal table, which our model never sees.

The out-of-distribution test follows the X-TC protocol on Taskonomy-trained models evaluated on Replica and ApolloScape. It compares the ROC-AUC of our ordering variance against consistency energy's reported 0.95.

Three outcomes would decide the matter. The idea is supported if three things hold together: the full method beats both the $\lambda=0$ and L1-agreement ablations on partially labeled NYU-v2 beyond seed variance; the information-use gap stays well above zero, showing no collapse to independent heads; and the ordering variance detects out-of-distribution inputs at least as well as consistency energy at a fraction of its cost. The idea is refuted if the $\lambda=0$ ablation matches the full method, or if raising $\lambda$ improves coherence only by driving the conditionals toward the marginals. In either case the honest conclusion would be that conditional auxiliary heads, not probabilistic duality, carry the benefit.