Spending Attention Where Pseudo-Labels Disagree With Geometry

When a segmentation model is trained on its own guesses for unlabeled images, the guesses are wrong in predictable places: along object boundaries, on thin structures, and where two objects meet. Efficient vision transformers have the same blind spot for a different reason. To avoid the cost of attention, which grows with the square of the number of image patches, they spend fine-resolution patches only in "important" regions and merge everything else into coarse patches. When those importance maps are learned from noisy self-generated labels, nothing prevents the model from spending its fine detail on its own confident mistakes. This proposal treats the two problems as one. A class-agnostic segmenter, a model that outlines physical objects without knowing what they are, can tell us where the teacher's labels conflict with visible object extents. That single disagreement signal decides where the student network spends fine patches and where it trusts the pseudo-labels.

In concrete terms, we run the Segment Anything Model offline once per unlabeled image and measure how mixed the teacher's labels are inside each segment. That mixedness score supervises a small head in a convolution-plus-attention student. The head decides which coarse patches to split into fine ones, and the same score down-weights the training loss where labels are unreliable. At test time the student runs without the segmenter and spends attention on predicted trouble spots. The question is whether this beats learned importance maps and confidence-based allocation at the same compute, on high-resolution semi-supervised benchmarks.

Core Claim

Under self-training, an importance map learned from the task loss will drift toward pseudo-label errors. A map supervised by disagreement between geometry and semantics will not. At matched FLOPs, a student whose fine tokens follow this disagreement map should reach higher mIoU and put a smaller share of its fine tokens on pseudo-label errors than one using learned significance in the style of SG-Former or teacher-entropy splitting.

The file flags this interaction as unstudied. SG-Former trains its significance map only on clean supervision. Our contribution is to measure the failure and to remove it with an external supervision signal for the importance map.

We deliberately do not claim that geometry-based label cleaning is new. Cleaning pseudo-labels with SAM is established: SAMST filters a Swin teacher's predictions, turns them into prompts, and lets SAM snap them to edges. Segment Anything is A Good Pseudo-label Generator shows SAM produces strong labels from clean scribbles. That same paper is a constraint on our design: re-prompting SAM from noisy class activation maps made labels worse, because SAM converts blurry errors into crisp false masks. We therefore never prompt SAM from teacher outputs. We use its unprompted segments only as a reference to compare against.

Research Program

The pipeline has a teacher, a fixed segment map, a disagreement score, and a student.

The teacher is a standard supervised segmenter trained on the labeled split. It produces per-pixel pseudo-labels $\hat y_p$ on unlabeled images. SAM (Segment Anything) runs once per unlabeled image in automatic mode and yields segments $s$. For each segment we compute its impurity:

$$u_s = 1 - \max_c \frac{1}{|s|}\sum_{p \in s} \mathbb{1}[\hat y_p = c],$$

which is the fraction of the segment's pixels that disagree with its majority teacher label. A high $u_s$ means one of two things. Either the teacher's boundary cuts through a physical object, which is an assignment error in the taxonomy of Combating Noise. Or SAM's segment leaked across two objects. In both cases the region is ambiguous and boundary-heavy, so it deserves both fine resolution and lower label trust.

We rasterize $u_s$ to the coarse patch grid as a target map. Low-impurity segments also receive majority-vote relabeling. This is the conceded, known cleaning step, and every baseline gets it as well.

The student is a hybrid network. A convolutional stream runs at 1/4 resolution and carries boundary detail at linear cost. A global attention stream operates on a quadtree of patches. Every 32-pixel patch starts as one token, and a lightweight split head decides whether to replace it with 8-pixel tokens. Under a split fraction $\rho$ with $N_c$ coarse patches, global attention costs on the order of $\big((1-\rho)N_c + 16\rho N_c\big)^2$ rather than $(16N_c)^2$. The factor 16 is the number of 8-pixel children per 32-pixel patch. The two streams exchange information through bidirectional cross-attention, adapted from LookupViT. LookupViT showed that bidirectional exchange beats unidirectional exchange by 8.5% on ImageNet, but it was never evaluated on dense prediction. Our dense setting is therefore also a test of its compressed-token design at pixel boundaries.

The split head is trained to regress the rasterized impurity map, and splits are chosen top-$\rho$ by its prediction. The pseudo-label loss at each pixel is weighted by $1 - u_{s(p)}$, in the spirit of the uncertainty-weighted soft targets of Combating Noise. At inference neither SAM nor the teacher is used.

We will also build the diagnostic that motivates the claim. Cityscapes and Potsdam are fully annotated, so the ground truth of the "unlabeled" images can be held back. That lets us measure exactly which pixels carry wrong pseudo-labels, and what share of each allocator's fine tokens lands on them over the course of training.

Baselines

The naive reference is the same student backbone trained on the labeled split only, with uniform coarse tokens, at the same FLOPs. A second naive point is plain self-training with hard teacher labels and SAM majority-vote cleaning but no split head, with the token budget spent as uniform medium-resolution patches. These two points tell us how much of any gain is just pseudo-labels plus the conceded cleaning step.

The strongest existing allocation method is learned significance-guided token reallocation as in SG-Former. It reported 52.7 mIoU on fully supervised ADE20K, +3.0 over Swin-B, at lower FLOPs. It is the direct competitor because it is exactly the "spend tokens where it matters, learned end to end" design whose behavior under noisy labels we are questioning. To compare fairly, we transplant its significance module into our student to drive the same quadtree splits. It is trained on the identical cleaned pseudo-labels with the same $\rho$, schedule, and tuning sweep.

The strongest existing semi-supervised result in the file on our primary benchmark is Auto-Annotation Quality Prediction. It reached 77.0% mIoU on Cityscapes with 30% manual labels plus filtered pseudo-labels, matching the 77.0% full-label baseline, but it used an 18-model ensemble. We will report against this number. We will also rerun its pixel-quality filter with a single-teacher input in place of our impurity weighting, so the label-trust component is compared at equal cost.

On Potsdam, SAMST reports only a 5.67% relative mIoU gain over its baseline at the 1/32 split. No absolute mIoU was found on Lacuna for SAMST. We will therefore rerun its prompt-and-refine pipeline to produce labels for our student and report absolute numbers ourselves.

Beating these baselines means two things. First, higher mIoU than the SG-Former-style allocator at equal FLOPs on both Cityscapes and Potsdam, across three seeds. Second, a measurably lower share of fine tokens on pseudo-label-error pixels.

Two ablations isolate the key ingredient. One drives splits by teacher entropy instead of SAM-derived impurity. The other decouples the two roles of the score, using impurity for splitting only or for loss weighting only.

Why not just…?

Why not just clean labels by majority vote inside SAM segments and train any efficient segmenter? This is the most serious threat. If our method's gain came from cleaning, this baseline would match it, and that is why we gave the voting step to every baseline. What the voting cannot do is decide where to spend compute at test time. If the uniform-token and SG-Former students trained on the same voted labels match ours at equal FLOPs, the claim fails, and we will report that.

Why not just weight the loss and choose splits by teacher confidence? Confidence is cheaper and needs no SAM. However, Combating Noise found that noise concentrates in a threshold "danger zone" and includes confidently wrong regions that confidence cannot flag. The entropy-split ablation tests this directly. If it ties our method, SAM adds nothing beyond cleaning.

Why not just run the SAMST pipeline and then a full-resolution student? This gives clean labels and fine tokens everywhere, at full quadratic cost. We include it as a compute upper bound. Success for our method means approaching its mIoU at a fraction of its FLOPs, not exceeding it.

Why not just use a pure CNN, which may tolerate label noise better? A Guide to Robust Generalization found that large ConvNeXt models beat transformers in robustness. We will include a parameter-matched ConvNeXt student trained on the same labels. This tells us whether attention allocation matters at all in this regime.

Proposed Experiments

The primary benchmark is Cityscapes semi-supervised at 1/8 and 30% labeled splits. Its high resolution makes quadratic cost real, and the 77.0% reference exists. The second is Potsdam at 1/32, following the SAMST protocol. PASCAL VOC 2012 serves as a lower-resolution check where SAM-based pseudo-labels are well studied; the scribble-trained DeepLab-v2 reached 76.6% test mIoU in Segment Anything is A Good Pseudo-label Generator, though under a different supervision regime. We report mIoU, boundary F-score, FLOPs, and measured latency at full resolution, sweeping $\rho \in \{0.05, 0.1, 0.2\}$ to trace accuracy–compute curves.

Three checks decide the claim. First, the drift diagnostic: under self-training, the SG-Former-style allocator's fine-token share on pseudo-label errors should rise over epochs while ours stays flat. If neither drifts, the motivating premise is false. Second, the matched-FLOPs comparison on the accuracy–compute curve against the SG-Former-style and entropy allocators. Third, an analysis of when impurity reflects SAM leakage rather than teacher error, done by checking held-back ground truth inside high-$u_s$ segments, so we know which role the score is really playing. As a secondary probe, the split head's behavior on PartImageNet will show whether part-whole granularity mismatch inflates impurity on correctly labeled objects. The relevant references there are SegFormer at 65.27% in PartImageNet and WPS-SAM at 68.93% in WPS-SAM.