Cheap, imprecise annotations are easy to get for semantic segmentation. Cityscapes ships 20,000 coarsely annotated street images next to its 5,000 finely annotated ones (The Cityscapes Dataset). Coarse labels are wrong in a predictable place: near object boundaries and on thin structures, where the annotator's polygon spills over or falls short. This note proposes training a real-time segmentation network on such labels by attaching several temporary prediction heads that look at the image at different spatial scales. Each head uses dilated convolution, a convolution whose taps are spread apart so it covers a wider area without extra parameters. Where heads that see different amounts of context disagree with a pixel's label, that pixel's loss is softly turned down. After training the heads are thrown away, so the deployed network costs exactly what it did before.
Put plainly, there are two claims. Disagreement between heads with different receptive fields is a better detector of boundary-structured label noise than disagreement between identical heads or than a single model's own loss. And using that disagreement as a per-pixel training weight lets a fixed, single-pass network gain from cheap coarse data without paying the multi-model inference cost that ensembles usually carry.
Core Claim
Collaborative Learning for Deep Neural Networks trains several heads on a shared backbone and has them distill into one another. It drops the heads at deployment and reports resistance to heavy label noise. Its heads are architecturally identical, though, and they vote uniformly. Their diversity comes only from random initialization, and nothing in the method uses their disagreement to weight individual training examples. Multi-scale dilated designs such as PSConv, CFPNet and RegSeg bake several dilation rates into one static network. They treat scale purely as a feature-extraction device and never as a source of evidence about which labels to trust.
The hypothesis joining these lines is that heads with different receptive fields make structurally different errors. A small-dilation head relies on local texture and edges. A large-dilation head relies on layout and context. A coarse label that has spilled across a boundary is contradicted by local evidence while context may still support it, whereas a genuinely ambiguous region tends to confuse both kinds of head together. Scale-structured disagreement should therefore separate "label is wrong" from "pixel is hard" better than disagreement among clones. Making that separation matters, because the known failure of loss-based reweighting is the self-reinforcing loop flagged in the robust-weighting literature (Robust Learning through Adaptive Instance and Update Weighting): a model downweights hard but valid patterns and never learns them.
Research Program
The base network is PIDNet-S (PIDNet), chosen because it is a strong real-time segmenter with a published speed. During training we attach $H=4$ lightweight auxiliary heads to the last shared stage. Each head is a two-layer $3\times3$ convolution stack with a single dilation rate from $\{1, 2, 4, 8\}$.
Every head, and the main PIDNet output, is trained with a per-pixel weighted cross-entropy. The weight for head $h$ at pixel $i$ is computed only from the other heads, a leave-one-out rule that keeps a head from certifying its own mistakes. Let $q_{-h}(y_i \mid x)$ be the mean probability that heads other than $h$ assign to the annotated class $y_i$. The weight is
$$ w_{h,i} = \min\!\left(\beta,\; \frac{q_{-h}(y_i\mid x)^{\gamma}}{\frac{1}{|B|}\sum_{j\in B} q_{-h}(y_j\mid x)^{\gamma}}\right), $$
In this rule $B$ is the set of coarse-labelled pixels in the minibatch, $\gamma \ge 0$ controls how sharply disagreement is punished, and $\beta$ caps any single pixel's influence. The weights are normalized to mean one within each batch. This prevents the collapse of all weights toward zero that the robust-weighting literature warns about. It plays the same role as the simplex constraint in FedLAW, which learns server-side weights over clients rather than pixels. The weights are treated as constants in the backward pass. Fine-labelled pixels always get weight one.
To further limit the feedback loop, training begins with a warm-up on fine labels only, after which coarse data is mixed in. The heads also exchange soft labels as in Collaborative Learning, so the one change being tested is where the weights come from.
The analysis has two parts. On data where the true noise is known, we measure how well each disagreement signal ranks corrupted pixels above clean ones. On real Cityscapes coarse labels, we measure whether downweighted pixels concentrate near true boundaries, using the fine annotations of the same images where both label sets exist. If disagreement among same-scale heads ranks noise as well as scale-diverse disagreement does, the core claim is false. The result would then reduce to Collaborative Learning plus reweighting, and we would report it as such.
Baselines
The naive reference is PIDNet-S trained on fine labels only, and the same network trained on fine plus coarse labels with plain unweighted cross-entropy. The PIDNet paper reports 78.6% mIoU on Cityscapes for PIDNet-S at 93.2 FPS on an RTX 3090. We will retrain it ourselves under our schedule and report both the cited and the reproduced number, so that gains are measured against our own reproduction.
The strongest accuracy competitor is FADC applied to PIDNet (Frequency-Adaptive Dilated Convolution), which adapts dilation per pixel at test time and reports 81.0 mIoU on Cityscapes at 37.7 FPS. It answers the question "can adaptivity buy accuracy?" by spending inference compute, which is the opposite trade from ours. Matching it in accuracy is not required. Beating it means at least matching its mIoU on boundary-heavy classes while running at PIDNet-S speed. Closing a meaningful share of the 78.6-to-81.0 gap at roughly 2.5 times its frame rate would be a positive outcome.
The direct method competitor is Collaborative Learning applied to the same PIDNet-S trunk with four identical heads. It gets the same coarse data, the same head count, the same training epochs and the same tuning budget for its loss weights. Its uniform-consensus labels serve as soft targets, with no per-pixel weights.
Two ablations isolate the key ingredient. The first keeps our weighting rule but makes all four heads use the same dilation rate, which separates scale diversity from simply having more heads. The second computes weights from the main head's own loss using a Student-$t$-style downweighting of large residuals, in the spirit of Robust Graphical Modeling with t-Distributions. This separates external disagreement from self-assessment.
As an upper reference we also report the four scale heads ensembled at test time. This shows what the ensemble would give if we paid for it. All methods share data, augmentation, input resolution, number of training iterations, and a fixed tuning budget of the same number of trials over their respective weighting hyperparameters.
Why not just…?
Why not just train on the fine labels and ignore the coarse ones? That is the strongest do-nothing option, and it may win if coarse labels are mostly harmful. The proposal only has value if weighted coarse training beats fine-only training, so this comparison is the first table. No result in the file reports PIDNet-S trained with Cityscapes coarse labels. There is no reported result found on Lacuna for that configuration, so we must measure it.
Why not just downweight high-loss pixels with a single head? This is the most serious threat. It is simpler, has no extra heads and costs nothing at inference. Our concern is the feedback loop: thin poles, sign boundaries and rare classes have high loss because they are hard, not because they are mislabelled, and self-weighting suppresses exactly those. The Student-$t$ ablation tests this directly. If it matches our method on mIoU and on noise-ranking quality, the multi-head machinery is unjustified and we will say so.
Why not just distill a multi-scale ensemble into one network? Offline distillation needs a fully trained teacher ensemble first. Collaborative Learning reports that its four-head joint training uses about 1.5 times the memory of one model versus about 4 times for a four-model ensemble. More importantly, a teacher's soft labels fix the target but say nothing about which ground-truth pixels to distrust. We will nonetheless include a distillation run from the test-time-ensembled scale heads into PIDNet-S, with matched total training compute.
Why not just bake multi-scale context into the trunk with PSConv or RegSeg-style blocks? These improve features at no inference cost; PSConv reports 21.1% versus 22.8% ImageNet top-1 error on ResNet-50. They are orthogonal to label trust. A static multi-scale trunk still fits wrong coarse labels at full weight. We treat them as compatible: one run swaps a PSConv-style trunk in, to check that our gains add to theirs rather than duplicating them.
Proposed Experiments
The first experiment is controlled. Fine Cityscapes training labels are corrupted with boundary erosion and dilation of $k$ pixels at several values of $k$, plus random class flips inside connected regions. Because the truth is known, we can measure the AUROC of each weighting signal for detecting corrupted pixels: scale-diverse, same-scale, self-loss and uniform. We also measure final validation mIoU. The decisive check is whether scale-diverse disagreement beats same-scale disagreement specifically on boundary corruption while tying it on class flips. That pattern is what the mechanism predicts. A uniform gap, or no gap, would undermine the scale story.
The second experiment is realistic: fine plus the 20,000 coarse Cityscapes images. We report mIoU on validation and on the held-out test server, a boundary F-score within a narrow band around true edges, and per-class IoU for thin classes. FPS and parameter count are measured on one GPU for every method, so the claim of equal inference cost is checked rather than asserted.
The third experiment tests transfer to CamVid. There, PIDNet reports 80.1% mIoU for PIDNet-S, and we inject synthetic coarse-style noise to see whether the chosen $\gamma$ and $\beta$ carry over without retuning, which addresses the hyperparameter sensitivity noted for simplex-constrained weighting.
The proposal succeeds if three things hold. Weighted coarse training beats both fine-only and naive coarse training. Scale-diverse heads beat same-scale heads and self-loss weighting on boundary noise. And the deployed network runs at the speed of unmodified PIDNet-S.
References
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Collaborative Learning for Deep Neural Networks
- PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer
- CFPNET: Channel-Wise Feature Pyramid For Real-Time Semantic Segmentation
- Rethinking Dilated Convolution for Real-time Semantic Segmentation
- Robust Graphical Modeling with t-Distributions
- PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers
- Byzantine-Robust Federated Learning with Learnable Aggregation Weights
- Frequency-Adaptive Dilated Convolution for Semantic Segmentation