Every dense vision model makes a choice at every pixel: how large a neighborhood to look at before deciding what that pixel is. Look too narrowly and a patch of white wall is indistinguishable from sky. Look too widely and the features for a chair leg absorb the floor next to it, which blurs the edge. Multi-scale networks handle this by computing features at several receptive-field sizes and blending them. The blending weights are usually fixed, or come from an attention map, and the blend itself happens in feature space. That means a boundary pixel's representation is a weighted average of a sharp local view and a polluted wide view. We propose to treat "how much context does this pixel need, for this task" as a hidden discrete variable. The network predicts separately under each context hypothesis and learns a posterior over which hypothesis explains the label. Tasks share a prior over this variable: if depth wants fine context at a pixel, the boundary task probably does too.
Put simply, the model keeps one clean prediction per context size instead of averaging features across sizes. It then infers, per pixel and per task, a probability distribution over which size to trust, with the tasks informing one another's choice. The model is trained with variational inference, which fits an approximate posterior by maximizing a lower bound on the data likelihood. This yields three things together: sharper boundaries, a readable map of context need, and calibrated uncertainty where context is ambiguous. We test all three on the standard multi-task dense benchmarks NYUD-v2 and PASCAL-Context.
Core Claim
The spatial context trade-off shows up as a fixed optimum in many settings. On the Materials in Context Database, Material recognition in the wild with the Materials in Context Database finds the best patch is 23–32% of the image, but only after a grid search. At the other extreme, Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs and A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning show that oversized windows pull in neighboring-class evidence and hurt accuracy.
These results motivate our first claim: the right context size is a per-pixel, per-task quantity. They also point at a mechanism. Contamination happens when wide-context features are mixed into a boundary pixel's representation.
Our central claim is that marginalizing over discrete context hypotheses at the output is better than blending context in feature space, especially at object boundaries. A coupled cross-task prior over those hypotheses is what makes the multi-task setting help rather than hurt. The second part is the multi-task contribution. MTI-Net showed that task affinities depend on scale: depth discontinuities align with semantic edges locally, while object-level shape relates to semantic category globally. We encode that finding as a prior over the joint context choice of the tasks, rather than as deterministic attention.
Model
Let $x$ be the image. Let $i$ index pixels, $k$ index tasks, and $s \in \{1,\dots,S\}$ index context hypotheses. Hypothesis $s$ is a light prediction head for task $k$ that reads the backbone up to stage $s$, with a full-resolution skip connection. Its receptive field therefore grows with $s$, and it never mixes in deeper, wider features. Let $z_{k,i}$ be the latent context choice. The likelihood is $p(y_{k,i}\mid x, z_{k,i}=s) = h_{k,s}(x)_i$.
The prior $p_\psi(z \mid x)$ has two couplings. The first is a cross-task mixture term in the style of the Gumbel-Softmax task prior of Variational Multi-Task Learning with Gumbel-Softmax Priors: task $k$'s prior at pixel $i$ is a learned mixture of the other tasks' posteriors, with mixing weights that differ per scale. The second is a spatial smoothness term: context need varies smoothly across a region and jumps at edges.
The approximate posterior $q_\phi(z\mid x,y)$ is mean-field across tasks and pixels. It is amortized by a small network that sees image features and training labels. We maximize
$$\mathcal{L} = \sum_{k,i} \mathbb{E}_{q_\phi(z_{k,i})}\big[\log h_{k,z_{k,i}}(x)_i\big] - \mathrm{KL}\big(q_\phi(z\mid x,y)\,\|\,p_\psi(z\mid x)\big).$$
At test time labels are unavailable, so we predict with the prior's mixture, $\sum_s p_\psi(z_{k,i}=s\mid x)\, h_{k,s}(x)_i$. The entropy of $p_\psi(z_{k,i}\mid x)$ is a per-pixel measure of context ambiguity.
We should be honest about when the variational machinery is necessary. With independent tasks and no spatial coupling, the exact marginal likelihood per pixel is a cheap sum over $S$ terms. In that case the model is a mixture of experts trained by maximum likelihood, and no variational bound is needed. The cross-task and spatial couplings make the exact posterior intractable, and they are what the ELBO buys us. This is why our ablations separate the two.
The model also has limits. With mean-field $q$, we expect underestimated posterior variance, a known weakness of mean-field noted in Zoubin Ghahramani. We also expect a risk that the prior collapses to a single scale. We monitor the per-scale usage entropy and anneal the KL weight.
Relation to Prior Work
Three deterministic decoders resolve context and task interaction in feature space. MTI-Net distills tasks at four resolutions and propagates coarse to fine. DeMT learns deformable sampling offsets per task. Inverted Pyramid Multi-task Transformer (InvPT) upsamples through joint cross-task attention. All three fuse scales before prediction and give no uncertainty. InvPT also reports persistent competition between semantic and boundary tasks, which we read as consistent with feature-space blending harming boundaries. We reuse their backbones and benchmarks, and we change where fusion happens and what is inferred.
Variational Multi-Task Learning with Gumbel-Softmax Priors supplies the cross-task prior technique. It showed gains in the low-data regime: 58.3% versus 50.4% for a deterministic multi-task baseline on Office-Home at 5% of the data. However, its latents are image-level features and classifiers with no spatial extent. We move the latent to a per-pixel context choice and make its task coupling depend on scale.
Deep Exponential Families motivates the idea of latent hierarchies in which higher levels shape lower ones, but it was only applied to text and ratings. We do not adopt its generative structure.
Pixel-wise Deep Learning for Contour Detection is the clearest evidence that multi-scale features matter for edges: concatenating Conv1–5 gives ODS 0.741, against 0.699 for Conv2 alone. Its fusion is static concatenation, which is exactly what we replace.
Research Program
We build the decoder on the InvPT backbone so that comparisons isolate the fusion mechanism, using $S=4$ context heads per task. The work proceeds in three steps.
The first step is diagnostic. We test whether inferred context maps are meaningful. The posterior should put fine hypotheses near ground-truth boundaries and coarse ones in textureless regions, with rank correlation to distance-to-edge as the measure. As a single-task sanity check on MINC, we also test whether the mixture's expected receptive field lands near the 23–32% optimum that the original authors found by grid search.
The second step is the main comparison on NYUD-v2 and PASCAL-Context. The third step is a low-data study on 10% and 25% subsets of NYUD-v2. This is where VMTL's evidence suggests a structured prior should matter most.
Baselines
The naive reference is independent single-task networks on the same backbone. The multi-task gain $\Delta_m$ is measured against these, following the protocol in Multi-Task Learning for Dense Prediction Tasks: A Survey.
The strongest existing method is InvPT. On NYUD-v2, Multi-task Learning with 3D-Aware Regularization reports InvPT at 53.56 mIoU and 0.5183 depth RMSE, and reports its 3D-regularized variant at 54.87 mIoU and 0.5006 RMSE. We rerun both under our training budget and do not copy these numbers. Beating InvPT means exceeding its rerun mIoU and $\Delta_m$ at matched FLOPs, together with a clear gain in boundary-band metrics.
We also compare against MTI-Net, the closest multi-scale deterministic design, which reports +10.91% on NYUD-v2 and +1.36% on PASCAL-Context. We further include DeMT, which reports $\Delta_m$ of 3.36% and 46.36 mIoU with Swin-T on NYUD-v2. For both we use the same backbone family, schedule, augmentation and loss-weight grid.
Two ablations isolate the key ingredients. The first is a feature-blend ablation: the same heads and backbone, with a per-pixel softmax over scales applied to features before a single head. This tests output-space against feature-space fusion. The second is a deterministic output mixture: our exact architecture trained by per-pixel maximum likelihood with independent tasks and no spatial or cross-task prior. This tests whether the variational coupling adds anything.
Every method gets the same data, the same grid of 12 loss-weight configurations, and three seeds. We report means with spread.
Why not just…?
Why not just use per-pixel softmax attention over scales, as MTI-Net-style decoders already do? This is the most direct competitor, and the feature-blend ablation is built to answer it. Our hypothesis is that averaging a wide-context feature into a boundary pixel reintroduces the contamination documented by Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs. If feature-blend matches our model within seed noise in the 5-pixel boundary band, that hypothesis is wrong.
Why not just train a deterministic mixture of experts over context sizes and skip the variational part? This is a real threat, and it could plausibly match us on full-data accuracy. The variational coupling is justified only if it helps somewhere measurable: in the low-data subsets, in cross-task consistency at edges, or in calibration. If the deterministic mixture matches on all three, we will report the contribution as output-space context selection alone and drop the variational claim.
Why not just add heteroscedastic uncertainty outputs, MC dropout, or ensembles for calibration? These give uncertainty but say nothing about why a pixel is uncertain, and they do not change the prediction mechanism. We compare expected calibration error and depth negative log-likelihood against uncertainty-weighted heads and a 3-model ensemble at matched compute. We expect ensembles to be competitive on calibration. Our claim is only that ours gets comparable calibration together with the accuracy and interpretability gains at no extra cost.
Why not just tune loss weights or add the 3D regularizer? The survey above found that grid search often beats automated balancers, so we give every method the same grid. The 3D regularizer is complementary and adds no inference cost. If budget allows, we stack it on our decoder rather than treat it as a rival.
Proposed Experiments
On NYUD-v2 we evaluate segmentation mIoU, depth RMSE, normal mean angular error, and boundary ODS. On PASCAL-Context we add part mIoU and saliency. We report $\Delta_m$ throughout, plus segmentation mIoU restricted to 5- and 10-pixel bands around ground-truth edges, calibration error, and depth likelihood. The file contains no reported boundary-band or calibration numbers for these methods on NYUD-v2, so all such comparisons come from our reruns.
Three checks are decisive. The first is whether our model beats the feature-blend ablation in the boundary band while matching it in region interiors. That pattern is the signature of output-space fusion avoiding contamination. The second is whether the coupled prior beats the deterministic mixture at 10% data by more than it does at full data. The third is whether the inferred context map correlates with distance-to-edge and, on MINC, recovers the known patch-size optimum without a grid search. A positive result on the first check with a negative result on the second would still be publishable, but as a narrower paper about where fusion should happen.
References
- Material recognition in the wild with the Materials in Context Database
- Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs
- A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning
- MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning
- Variational Multi-Task Learning with Gumbel-Softmax Priors
- Factorial Learning and the EM Algorithm
- DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction
- Inverted Pyramid Multi-task Transformer for Dense Scene Understanding
- Deep Exponential Families
- Pixel-wise Deep Learning for Contour Detection
- Multi-Task Learning for Dense Prediction Tasks: A Survey
- Multi-task Learning with 3D-Aware Regularization