Multi-task learning trains one shared network on several prediction targets at once, such as diagnosis, mortality, and length of stay from the same patient record. Training them together often helps each target. The tasks can also fight, pulling the shared weights in opposite directions or letting the task with the largest gradients dominate. The best-known fixes, such as RotoGrad's rescaling and rotation of per-task gradients, quietly assume that every training batch carries a label for every task. Real multi-task data rarely does. In clinical records, sensor networks, and partially annotated vision datasets, labels and whole input modalities are absent for many examples. Often they are absent for reasons tied to the input itself, as when a test is ordered only for sicker patients.
This note proposes to study and repair gradient balancing under missing labels. We claim that existing balancing methods become noisy and biased when the set of observed tasks changes from batch to batch and depends on the input. A simple correction should restore their benefit. The correction has two parts. The first keeps per-task gradient statistics across batches instead of recomputing them on whatever subset happens to appear. The second reweights each observed task's gradient by the inverse of its estimated probability of being labeled.
Each task is written once as a declared output site. A per-batch record of which labels exist then determines whether that site contributes a gradient. This borrows the "observe if present, otherwise predict" mechanism from probabilistic programming. The practical payoff is that the same model can run under every balancing policy without code changes, which keeps the comparisons fair.
Core Claim
Two lines of work each solve half of this problem. RotoGrad tackles task conflict in two ways. It rescales task gradients toward a convex combination of magnitudes that favors slow-converging tasks, and it learns a rotation $R_k$ per task so that task-specific views of the shared feature align in gradient direction. Its training loop needs a fresh gradient from every task at every step. MultiModN handles missing inputs by passing each available modality through its own encoder into a running state vector and skipping absent ones. It matches parallel-fusion AUROC on MIMIC-IV, improves AUROC by 20% on Weather2k under heavy sensor missingness, and resists missing-not-at-random bias. It does no gradient balancing at all.
The research file names this tension explicitly. Joint gradient homogenization needs all task gradients, and modular skipping guarantees you will not have them. Our claim is that the tension can be resolved at the level of the gradient estimator rather than the architecture.
Concretely, let $m_k(x)\in\{0,1\}$ indicate whether example $x$ has a label for task $k$, and let $\pi_k(x)$ be the probability that it does. A balanced update that is unbiased for the full-label objective is
$$ \hat g = \sum_k \lambda_k \, \frac{m_k(x)}{\hat\pi_k(x)} \, R_k^\top \nabla_{r_k} \mathcal{L}_k , $$
where:
- $\lambda_k$ is the balancing weight;
- $r_k = R_k z$ is the rotated shared feature for task $k$;
- $\mathcal{L}_k$ is the task loss;
- $\hat\pi_k$ is a small learned model of label availability.
The balancing weights $\lambda_k$ and the rotation updates are computed from exponential moving averages of each task's gradient norm and direction, updated only on batches where the task appears. This prevents the normalizer from jumping whenever the observed subset changes.
The approach makes a falsifiable prediction. When labels are missing completely at random, $\hat\pi_k$ is constant, the reweighting does nothing, and simple loss masking should already work. The correction should matter specifically when missingness depends on the input.
Research Program
The first piece is a small training library in which a multi-task model is specified declaratively. Shared encoders, optional per-modality encoders, and named task output sites are declared once. Execution policies are swappable handlers: equal weighting, RotoGrad, and our partial-label variant.
The design borrows directly from ProbFX. ProbFX maps variable names to values in a "model environment" and treats a variable as observed if present and sampled if absent. Composed handlers in Effect Handlers for Programmable Inference then derive different inference algorithms from one model. Both works stay with symbolic Bayesian models and never touch deep multi-task optimization. We reuse their separation of concerns, not their language.
This plays the same role that AutoMTL plays for architecture. AutoMTL compiles an unmodified CNN into a search space over sharing choices but leaves loss balancing to external routines and resolves everything statically. We make no claim that this software layer is itself a scientific contribution. Its job is to let one model definition run under every policy with identical data, seeds, and budgets.
The second piece is the estimator above. It comes with an analysis of the bias of naive masking. Under input-dependent missingness, the subset of tasks present in a batch correlates with the inputs. Each rotation $R_k$ therefore aligns to a skewed slice of the feature distribution, and the magnitude normalizer is computed over a changing set of tasks. We will measure both effects directly. We will track the variance of $\lambda_k$ across steps and compare the cosine similarity between gradients in rotated space on observed-task subsets against a held-out fully labeled probe set.
We will also study the variance cost of inverse-probability weights when $\hat\pi_k$ is small. Clipping is the obvious remedy, and its effect will be reported rather than tuned away.
The third piece handles missing modalities. When a modality is absent, MultiModN-style skipping is used. The availability model $\hat\pi_k$ conditions on which modalities were present, so input missingness and label missingness are handled in one estimator.
Baselines
The naive reference point is masked hard parameter sharing. This is one shared encoder with task heads, equal loss weights, and losses for unlabeled tasks set to zero. It is what most practitioners would actually run.
The single-task reference trains one model per task on that task's labeled examples only. It never suffers from missing labels, so it is the right bar for asking whether sharing helps at all here.
The strongest balancing competitor is RotoGrad with loss masking, using its published magnitude and rotation updates computed over whatever tasks are present in each batch. It is the strongest competitor because it is the method our correction modifies. If it already degrades gracefully, our proposal is unnecessary.
The strongest missing-data competitor is MultiModN with equal task weights. It is the established method built for this data regime. On MIMIC-IV, the file reports only that its AUROC is "nearly identical to the P-Fusion baselines," and on Weather2k it reports a 20% AUROC improvement under missingness (MultiModN).
On fully labeled NYUv2 the strongest reported number is from DeMT: a 3.36% average per-task improvement over single-task baselines ($\Delta_m$) with a Swin-T backbone. That serves as our sanity anchor at 100% label availability. No reported result was found on Lacuna for any multi-task method on NYUv2 or MIMIC-IV with partially withheld task labels.
All methods share the same backbone, data split, and number of optimizer steps, and each receives the same hyperparameter search budget, for example 20 trials over learning rate and the method's own knobs. The baselines are run in the same declarative harness, so only the handler changes.
Two ablations isolate the key ingredients. The first keeps the moving-average statistics but drops the reweighting. The second keeps the reweighting but drops the moving averages. A third variant uses the true availability probability where it is known, as in synthetic missingness, to separate estimator error from method error.
Beating the baselines means two things. Under input-dependent missingness, the method must exceed masked RotoGrad and MultiModN on mean per-task $\Delta_m$ (vision) or mean AUROC (clinical and weather) by more than seed-to-seed spread over five seeds. It must also do so without losing to masked RotoGrad under completely random missingness.
Why not just…?
Why not just mask the losses and run RotoGrad unchanged? This is the most likely real threat, and it is a baseline. The argument against it is mechanistic. Its magnitude normalizer and rotation targets are recomputed over a different task subset every step, and under input-dependent missingness that subset is correlated with the inputs. If masked RotoGrad matches our method under the input-dependent missingness conditions, the proposal fails and we will report that.
Why not just build batches that contain every task? Complete-case batching restores RotoGrad's assumptions. However, it throws away every example with any missing label, which on clinical data can be most of the data. Under input-dependent missingness it also trains only on the subpopulation that tends to be fully labeled, which is exactly the bias MultiModN was designed to avoid. We will include complete-case RotoGrad as an additional run. We expect it to be competitive when missingness is mild and to fall behind as it grows.
Why not just fill in missing labels with a teacher model's predictions? Pseudo-labeling makes every batch look complete. Its errors are concentrated where labels are missing, which under input-dependent missingness is a systematically different population, so the balancer would align gradients to the teacher's mistakes. We will include it as a baseline, with a teacher trained per task on labeled data. If it wins, the honest conclusion is that imputation beats reweighting in this regime.
Why not just train separate single-task models? They are immune to label missingness, and on some tasks they may win. The case for sharing rests on evidence that joint training regularizes and helps in low-data regimes. MGM reports multi-task models at 50% data often exceeding single-task models at 100%. The single-task baseline tells us, per task, whether that holds here.
Why not just use a plain callback interface instead of an effect-style specification? For the empirical question, a callback API would serve equally well. We present the declarative layer only as the mechanism that guarantees identical model code across policies, not as a source of accuracy.
Proposed Experiments and Decisive Checks
The first experiment is a controlled stress test on NYUv2 (segmentation, depth, normals) and CelebA attributes. We withhold task labels at availability rates of 100%, 70%, 40%, and 15%, under two mechanisms. In the completely random condition, labels are dropped uniformly. In the input-dependent condition, the probability of dropping depth labels rises with scene clutter and the probability of dropping attribute labels depends on other attributes. We plot each method's gain over masked equal-weight sharing as availability falls.
The decisive check has two predictions. In the completely random condition, our method and masked RotoGrad should track each other. In the input-dependent condition, masked RotoGrad's advantage should shrink or reverse while ours holds. If the curves do not separate in the input-dependent condition, the core claim is wrong.
The second experiment uses natural missingness. On MIMIC-IV's ten clinical tasks and on Weather2k, we compare all methods using MultiModN-style modality skipping as the shared encoder. Mean AUROC is reported, along with worst-task AUROC, since balancing is supposed to protect neglected tasks.
The third experiment is diagnostic. On the synthetic splits, we log the step-to-step variance of the balancing weights. We also measure cosine similarity between task gradients on a fully labeled probe set. This tests whether gains come from the mechanism we describe or from incidental regularization.
References
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- MultiMoDN - Multimodal, Multi-Task, Interpretable Modular Networks
- Modular probabilistic models via algebraic effects
- Effects and Effect Handlers for Programmable Inference
- AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learning
- DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
- Generative Modeling for Multitask Visual Learning