A single network that predicts several things at once from one image, such as semantic segmentation, depth and surface normals, is cheaper than one network per task. Such a network is called a multi-task network: one shared trunk feeds a small output head for each task. It usually ends up worse on some tasks than the separate single-task networks it replaces. One known fix is knowledge distillation, meaning the compact student is trained to imitate stronger teacher models as well as the labels. Here the teachers are frozen single-task experts. Current methods apply that imitation signal with the same strength everywhere. That includes the depth teacher at an object boundary, where it is typically wrong, and in the middle of a flat wall, where it is almost certainly right. We propose to weight the distillation signal at every pixel and for every task by the inverse of the teacher's expected error at that location. The expected error is measured "out-of-fold", on images the teacher never trained on. The idea is the classical rule for fusing noisy measurements, which gives each source a weight inversely proportional to its variance, applied to a teacher's per-pixel reliability.
The point of doing this carefully is that the obvious version fails for a specific reason. A teacher that predicts its own uncertainty on its own training images will report near-zero error almost everywhere, because it has fit those images. The resulting weights carry no information. Our core claim is that cross-fitted variance turns inverse-variance weighting into a working per-pixel control on how much to trust the teacher. Cross-fitted means each training image's teacher variance comes from a teacher trained on the other half of the data. The same variances also give an analytic, non-learned scale for balancing tasks against each other. The approach stands or falls on one question: does this per-pixel information beat simply re-weighting whole tasks? The experiments are built around that comparison.
Core Claim
Three bodies of work meet here without overlapping. Distillation into multi-task students already works. Knowledge Distillation for Multi-task Learning trains frozen single-task experts, then aligns a multi-task student's shared features to them through small 1×1 adaptors. It reports 79.09% average accuracy on SVHN plus Omniglot against 76.40% for uniformly weighted multi-task training. Online Knowledge Distillation for Multi-Task Learning trains the teachers alongside the student and re-weights tasks by the ratio of multi-task to single-task loss. Both methods use one distillation strength per task, and neither uses any notion of where a teacher is reliable.
On the weighting side, Investigating Uncertainty Weighting for Multi-Task Learning and its precursor Analytical Uncertainty-Based Loss Weighting show two problems with the standard approach, where task weights are learned as noise variances. The weights adapt slowly, and the method over-weights easy tasks. They replace it with an analytic softmax over inverse task losses (UW-SO). UW-SO weights whole tasks, never individual examples or pixels, and involves no teacher. Instance-Level Task Parameters does weight per example and per task. It does so by storing a learned uncertainty parameter for every training sample, which gives no estimate for new data and no distillation.
The variance-weighting literature supplies the mechanism and its failure mode. Batch Inverse-Variance Weighting shows that naive inverse-variance weights diverge under stochastic gradient descent and that two fixes stabilize them: a variance floor and normalization within each mini-batch. That paper assumes the per-sample noise variances are known. We reuse its stabilization directly. Our contribution is a principled source of the variances: out-of-fold teacher error predictions.
The closest uncertainty-aware distillation work is Learning Task-Agnostic Representations through Multi-Teacher Distillation. It replaces mean-squared-error distillation with a Gaussian-mixture likelihood that learns the covariance around teacher embeddings. It produces frozen general-purpose features and does not train task heads or balance competing task gradients. Our setting is supervised dense multi-task training. Our uncertainty attaches to the teacher's task predictions rather than to embeddings.
The Objective
For task $k$ and output element $i$ (a pixel), we use the following quantities:
- $s_{ik}$ is the student's prediction.
- $y_{ik}$ is the label.
- $t_{ik}$ is the frozen teacher's prediction.
- $\ell_k$ is the task's usual loss: L1 for depth, cosine for normals, cross-entropy or KL for segmentation.
- $\hat v_{ik}$ is the teacher's predicted expected loss against the label at that element.
- $\bar r_k$ is the teacher's average out-of-fold loss on task $k$.
The student minimizes
$$\mathcal{L}=\sum_{k=1}^{K}\frac{1}{\bar r_k}\sum_{i\in\mathcal{B}_k}\Big[\ell_k(s_{ik},y_{ik})+\lambda\,\omega_{ik}\,\ell_k(s_{ik},t_{ik})\Big],\qquad \omega_{ik}=\frac{|\mathcal{B}_k|\,(\hat v_{ik}+\epsilon)^{-1}}{\sum_{j\in\mathcal{B}_k}(\hat v_{jk}+\epsilon)^{-1}}.$$
The remaining symbols are as follows:
- $\mathcal{B}_k$ is the set of labeled elements for task $k$ in the mini-batch.
- $\lambda$ is one global distillation strength.
- $\epsilon$ is the variance floor borrowed from batch inverse-variance weighting.
The design separates two effects that the literature mixes together. The weights $\omega_{ik}$ average exactly one over each batch, so they only redistribute distillation effort toward locations where the teacher is trustworthy. They do not change how much a task counts in total. The factor $1/\bar r_k$ handles cross-task balance. It divides each task's loss by the error level a strong single-task model actually achieves, which makes the tasks' losses dimensionless and comparable. The factor is fixed before student training, so it cannot have the slow adaptation that learned uncertainty weights suffer from. With this split, each part can be switched off on its own and the gain attributed.
There is a link to optimization theory. Knowledge Distillation Performs Partial Variance Reduction shows, for linear and deep linear models, that distillation acts like a control variate. It subtracts a correlated teacher gradient to cancel gradient noise. Training loss is U-shaped in the distillation weight, and the optimum depends on how good the teacher is. Our per-element $\omega_{ik}$ is the natural local version of that tuned weight. We treat this as motivation and a diagnostic, not a theorem, since that paper's proof covers only the single-task linear case.
Research Program
The first stage builds cross-fitted teachers. For each task we split the training set in two and train one single-task teacher per half. Each teacher has an extra lightweight head that predicts its own per-pixel expected loss. That head is fit only on the held-out half, using the residuals the teacher actually makes there. Every training image therefore gets a teacher prediction and a variance estimate from a model that never saw it. At distillation time, each image's target and $\hat v$ come from its out-of-fold teacher. We will also measure two properties of the variances. Calibration is the correlation between $\hat v$ and the realized error on a third held-out split. Collapse is the spread of $\hat v$ when it is computed in-fold instead.
The second stage trains the multi-task student with the objective above. It reuses the 1×1 adaptor feature alignment from Knowledge Distillation for Multi-task Learning as a fixed component, so that our change is confined to the output-level weighting.
The third stage is analysis:
- Where the gain comes from. We will check whether improvements concentrate in high-variance regions such as boundaries, thin structures and specular surfaces. We will also check whether they come at the cost of flat regions.
- Gradient noise. We will record gradient noise in the shared trunk under each weighting scheme, measured as the signal-to-noise ratio Knowledge Distillation Performs Partial Variance Reduction uses. This tests whether variance weighting actually reduces optimization noise.
- Student capacity. We will vary student capacity from a small encoder to ResNet-101. Analytical Uncertainty-Based Loss Weighting reports that weighting schemes stop mattering on large trunks. We expect the per-pixel benefit to shrink with capacity too, and want to find where it disappears.
Baselines
The naive reference is the multi-task network trained with equal task weights and no distillation. Every relative gain is reported against it and against independent single-task networks, using the standard average relative gain over all task metrics, Δm.
The strongest existing method is Online Knowledge Distillation for multi-task learning (Online Knowledge Distillation for Multi-Task Learning). It reports a 6.22% improvement over its baseline multi-task model on NYU-v2 and 9.19% on Cityscapes. On Cityscapes it lists Random Loss Weighting as the closest competitor at 7.69%. Those numbers are relative to that paper's own baseline and architecture. We will therefore rerun it in our codebase with the same trunk, data splits, epochs, and hyperparameter sweep size. Beating it means a higher Δm averaged over three seeds, with the gap exceeding seed-to-seed spread on both datasets.
The two-stage method of Knowledge Distillation for Multi-task Learning is the most direct ancestor. For NYU-v2 and Cityscapes the file gives only a qualitative claim of better mIoU and lower depth error, with no reported number found on Lacuna. We will rerun it in two versions. The first uses its original single teachers. The second uses our two cross-fitted teachers per task averaged together, with uniform distillation weight. The second version gives the baseline exactly our teacher compute, so any remaining gap is attributable to the weighting and not to having more teachers.
UW-SO from Investigating Uncertainty Weighting for Multi-Task Learning is run with its reported temperature for NYU-v2 (T=2). We also sweep the temperature on Cityscapes, because the file reports no temperature for that dataset. It is run in two forms: on task losses alone, and with each distillation loss treated as an extra task. The second form is the most dangerous competitor, because it is a cheap analytic weighting that already sees the distillation terms.
Two ablations isolate the key ingredient:
- Shuffled weights. The $\omega_{ik}$ values are randomly permuted among the elements of each batch, so the weight distribution is preserved but the location information is destroyed.
- In-fold variance. $\hat v$ comes from the teacher's head fit on its own training data, which tests whether cross-fitting matters.
We further report three switched-off variants: task scaling $1/\bar r_k$ only, per-pixel $\omega$ only, and both. All methods share the same hyperparameter budget: twelve configurations, chosen on a validation split carved from training data.
Why not just…?
Why not just apply UW-SO to the task and distillation losses together? This is the real threat. If whole-task re-weighting captures most of the benefit, per-pixel variance is unnecessary overhead. UW-SO is analytic and fast, and Investigating Uncertainty Weighting for Multi-Task Learning shows it avoids the slow adaptation and easy-task overfitting of learned weights. It cannot, however, tell a reliable wall pixel from an unreliable edge pixel in the same depth map. The test is the comparison between UW-SO on distillation losses and our objective with $\omega$ shuffled or intact. If intact $\omega$ does not beat the shuffled version and UW-SO by more than seed noise, we drop the per-pixel claim and report that task-level weighting suffices.
Why not just gate distillation by teacher confidence, such as softmax entropy? For segmentation, entropy is a reasonable stand-in and we include it as a baseline. Depth and normals produce no softmax, so entropy gives no reliability signal there. Even for segmentation, in-sample confidence on training images is inflated for the same reason in-fold variance collapses. Entropy gating also lacks a common scale across tasks. The in-fold ablation and the entropy baseline both test this. If entropy-gated segmentation distillation matches our segmentation numbers, we will say so and restrict the claim to regression tasks.
Why not just use equal weights on a big enough trunk? Analytical Uncertainty-Based Loss Weighting found that equal weighting approaches sophisticated schemes on ResNet-101. On the other hand, MoKD, which balances task and distillation gradients for a single task, still finds substantial gradient conflict on large vision transformers. The capacity sweep settles which regime holds here. Our expectation, and the claim we would defend, is a gain concentrated in compact students, which is the deployment case distillation exists for.
Why not just store per-sample uncertainty parameters as in Instance-Level Task Parameters? That method helps most under corrupted labels, reducing surface-estimation error by up to 60% on SURREAL under 40% poisoned labels. It needs one parameter per sample per task and has nothing to offer on unseen images. Our variances come from a model, but they target a different failure: uneven teacher error. They are not designed for label corruption. Where labels are wrong, out-of-fold residuals rise and our method distills less, which is the wrong direction for label-noise robustness. We run Instance-Level Task Parameters on clean NYU-v2 and Cityscapes as a baseline. We do not claim robustness to corrupted labels.
Proposed Experiments
The main experiments use NYU-v2 (segmentation mIoU, depth absolute and RMSE error, normal mean angle error) and Cityscapes (segmentation mIoU, disparity error). These are the benchmarks shared by every bridging paper in this space. We report Δm against single-task networks, per-task metrics, and three seeds.
The first decisive check is the shuffled-weight ablation. If location-specific weights do not beat permuted ones, the core idea is false.
The second is the in-fold versus out-of-fold comparison. If in-fold variance works equally well, cross-fitting is unnecessary and the method gets cheaper. If it fails, as we predict from variance collapse, that confirms why naive teacher-uncertainty weighting has not caught on.
The third is a stratified evaluation. We bin test pixels by the teacher's held-out error and check whether the student's gain over uniform distillation is largest in the high-error bins. We also check that the student does not lose accuracy in the low-error bins.
Finally, the capacity sweep and the gradient signal-to-noise measurements say whether the effect is an optimization effect that fades with model size. A positive result is a Δm gain over the matched-compute version of Knowledge Distillation for Multi-task Learning and over rerun online distillation on both datasets for compact students, with the shuffle control explaining the gain. A negative result, where UW-SO over distillation losses matches us, would still be worth publishing, because it would show that teacher reliability in multi-task distillation is a task-level quantity.
References
- Knowledge Distillation for Multi-task Learning
- Online Knowledge Distillation for Multi-Task Learning
- Investigating Uncertainty Weighting for Multi-Task Learning: Insights and Analytical Alternative
- Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning
- Instance-Level Task Parameters: A Robust Multi-task Weighting Framework
- Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
- Learning Task-Agnostic Representations through Multi-Teacher Distillation
- Knowledge Distillation Performs Partial Variance Reduction
- MoKD: Multi-Task Optimization for Knowledge Distillation