Teaching a Sparse Segmenter Where to Look: Routing by Predicted Teacher Disagreement

Fast segmentation networks often save compute by settling "easy" pixels early and spending the expensive layers only on the rest. The rest is usually object boundaries, since most pixels are road, sky, or organ interior. These networks decide which pixels are hard using their own early-layer confidence. That signal is known to fail in two ways. Early layers are confidently wrong on pixels they cannot recover. And a large share of the pixels that look hard against the ground-truth labels are hard only because the labels themselves are noisy.

We propose a different decision rule. A small head predicts, for each pixel, whether the cheap early prediction will disagree with a strong dense "teacher" network, meaning the full-compute model used for knowledge distillation (training a small model to imitate a larger one's soft outputs). Only pixels predicted to disagree get routed to the expensive sparse layers. There, the refinement is trained against the teacher's soft outputs instead of one-hot labels. The claim is that routing on predicted teacher disagreement buys more accuracy per unit of compute than routing on self-confidence or on predicted ground-truth error. The expected advantage is largest at object boundaries and under label noise, where the teacher smooths annotation errors that ground-truth-based routing would chase.

Core Claim

Sparse spatial convolution means running heavy convolution kernels only at a selected subset of pixel locations. Coarse-to-fine segmentation uses a cheap global pass to decide where fine computation goes. Both need a routing rule: a policy that says where to spend compute. In every system in our file, that rule is derived from the student itself.

None of these uses a teacher to shape where computation goes. The distillation work for segmentation has the mirror-image gap: it always distills into static dense students.

Our contribution is not to bolt a distillation loss onto a cascade. It is to change the routing target itself to "where would the teacher disagree with me." We then test whether that target carries information that neither the student's confidence nor its ground-truth error carries.

Concretely, let $\hat y_k(p)$ be the class predicted at pixel $p$ by early exit $k$, and let $y_T(p)$ be the teacher's predicted class. A lightweight head $g_k$ reads exit-$k$ features and outputs $g_k(p) \in [0,1]$. It is trained with binary cross-entropy toward the disagreement indicator:

$$ \mathcal{L}_{\text{route}} = \sum_p \mathrm{BCE}\big(g_k(p),\ \mathbb{1}[\hat y_k(p) \neq y_T(p)]\big). $$

A soft variant replaces the indicator with the normalized KL divergence between the exit's distribution and the teacher's distribution. At inference, pixels whose $g_k(p)$ is above a threshold chosen to meet a compute budget continue to later sparse stages. The rest exit.

On routed pixels, the later stages receive a pixel-wise KL loss toward the teacher, following the edge-region treatment of BPKD. They also receive a pairwise similarity loss computed only among routed pixels. This makes the relational term of Structured KD cost $O(K^2)$ in the number of active pixels $K$, rather than $O(N^2)$ over all $N$ pixels. It becomes affordable at full Cityscapes resolution.

Research Program

The primary testbed is Cityscapes. We build on the public ADP-C code (liuzhuang13/anytime) with an HRNet-W48 backbone, so that masking, interpolation, and exit heads are held fixed and only the routing rule and refinement targets change. For the teacher we use two options:

  • the fully dense HRNet-W48, which isolates the effect of the routing target without adding capacity;
  • a stronger dense segmenter, which tests whether a better teacher helps.

The work has three parts.

Routing heads and budget sweep. We build the disagreement-prediction heads and sweep the routing threshold to trace full accuracy-versus-compute curves, rather than reporting single operating points. This matters because the file documents that fixed-threshold cascades are sensitive to the threshold.

Diagnostics of each routing signal. For each signal we measure routing recall: the fraction of final-exit errors (against clean labels) that had been routed to refinement. We also measure the fraction of compute spent on pixels whose label is later shown to be noise.

Medical extension. As a secondary study, we port the same idea to NMSW on WORD and TotalSegmentator. There the NMSW importance distribution is supervised to predict where a dense sliding-window teacher disagrees with the coarse output. The question is whether teacher-shaped routing reduces the cascade false negatives that NMSW and Small Organ Segmentation in Whole-body MRI (which crops regions of interest and feeds stage-one probabilities as extra input channels) cannot recover.

Baselines

Naive references. The first is the dense HRNet-W48 run at full compute. ADP-C reports it at 80.7% mIoU and 696.2 GFLOPs on Cityscapes. The second is the same backbone with plain early exits, which ADP-C reports at a mean over exits of 62.7% mIoU at 816.0 mean GFLOPs. These fix the ceiling and the no-routing reference.

Strongest competitor: ADP-C. It is the closest system on the same benchmark, and it already uses mini encoder-decoder exit heads to stabilize early confidence. ADP-C reports 81.3% mIoU at the final exit with 387.1 GFLOPs (a 44.4% reduction), and a mean over exits of 65.7% mIoU at 195.6 mean GFLOPs. It also reports a 1.48x CPU wall-clock speedup. For a fair comparison, we rerun ADP-C from its codebase with the same backbone, training schedule, and augmentation. We give its confidence threshold the same tuning budget as ours. Beating it means a routing curve that dominates ADP-C's reproduced curve: higher mIoU at matched final-exit GFLOPs, or at least 15% fewer GFLOPs at matched mIoU. The gain should hold across three seeds and not shrink under measured wall-clock.

Deep Layer Cascade. Its reported Cityscapes numbers are not in our file; there is no reported result found on Lacuna for Deep Layer Cascade on Cityscapes. We therefore reimplement its fixed-threshold hard exits inside the same backbone.

Static distillation. Structured KD applied to a compact dense student gives the static-compression alternative. It reports MobileNetV2Plus rising from 68.9% to 74.0% mIoU.

Ablation isolating the key ingredient. The most important comparison replaces the teacher-disagreement target with the ground-truth error indicator $\mathbb{1}[\hat y_k(p) \neq y(p)]$, where $y(p)$ is the ground-truth label. Everything else stays identical, including the soft-target refinement losses. A second ablation keeps teacher routing but trains refinement on hard labels. Together these separate what the teacher contributes to deciding where to compute from what it contributes to deciding what to predict.

Why not just…?

Why not just calibrate the early-exit confidence? Temperature-scaling or tuning the threshold of ADP-C could fix some miscalibration cheaply. It cannot fix pixels that are confidently wrong, which is the failure Deep Layer Cascade describes as unrecoverable early commitment. We include a calibrated-ADP-C baseline with the same tuning budget. If it matches our curve, the claim fails.

Why not just train the router to predict the student's own ground-truth error? This is the real threat. It needs no teacher and targets errors directly. Our argument for the teacher rests on Deep Layer Cascade's finding that about 30% of extremely hard pixels are annotation artifacts. A router trained on ground-truth error will learn to spend compute on them. A router trained on teacher disagreement will not, provided the teacher smooths those labels. The ground-truth-error ablation and the noise-injection experiment below settle this directly. If the two routers tie even under injected noise, we would report that the teacher's value lies only in the refinement targets.

Why not just distill into a small dense network? Structured KD's best Cityscapes student reaches 74.0% mIoU. That is well below ADP-C's 81.3% at a fraction of dense cost. A static student also keeps spending compute on every background pixel. We will include this curve, but the evidence in the file already suggests it sits below spatially adaptive methods at high accuracy.

Proposed Experiments and Decisive Checks

Cityscapes curves. On Cityscapes validation we report the following for every method, over three seeds:

  • mIoU versus final-exit and mean GFLOPs;
  • boundary-band mIoU using the trimap metric BPKD reports;
  • measured GPU and CPU latency, since the file stresses that masked compute does not automatically become speed.

Decisive check 1: label-noise injection. We corrupt Cityscapes training boundaries by random dilation and erosion of object masks at controlled widths, and evaluate on clean validation labels. The prediction is that the ground-truth-error router's compute on corrupted pixels grows with noise while the teacher-disagreement router's does not, and that the mIoU gap widens with noise. If neither happens, the central claim is wrong.

Decisive check 2: routing recall at matched budget. Teacher-disagreement routing should capture more of the final-exit errors than self-confidence routing, because it is trained on a target the early layers cannot see from their own logits.

Semi-supervised extension. Structured KD reports a +13.5% mIoU gain from teacher soft targets on unlabeled CamVid data. Following that, we train the router on unlabeled frames, where ground-truth-error routing is impossible. This checks whether the teacher also extends routing supervision beyond labeled data.

Medical extension. On WORD and TotalSegmentator, the comparison is against NMSW at a matched compute budget. We report Dice, compute in trillions of multiply-accumulates, and per-organ small-structure recall. NMSW reports cutting compute from 88.0 to 8.00 trillion multiply-accumulates on WORD and exceeding sliding-window Dice on the TotalSegmentator organ task. The file gives no per-class Dice numbers for NMSW, so we reproduce it.