Rebalance Regions, Not Pixels, When Segmentation Labels Are Scarce

Semantic segmentation models learn by labeling every pixel. When most images come without labels, the usual remedy is self-training: a "teacher" network labels the unlabeled images and a "student" learns from those guesses, called pseudo-labels. Self-training breaks down when classes are long-tailed, meaning a few classes such as road, sky and building cover most pixels while many classes such as rider, train or a rare piece of furniture appear rarely. The teacher is unsure about rare classes, so the confidence filters that remove bad guesses also remove nearly every rare-class pixel. The usual repair, upweighting rare-class pixels in the loss, is also known to fail in segmentation. It makes the model paint rare classes over neighboring regions, and the resulting false positives lower mean intersection-over-union (mIoU), the standard per-class overlap score.

This note proposes moving the unit of rebalancing from the pixel to the soft region: for each image and class, a probability-weighted average of the student's features over the area the teacher believes belongs to that class. Building this average from soft teacher probabilities, with no hard threshold, keeps the fractional evidence from uncertain rare-class pixels that thresholding throws away. Counting regions instead of pixels also shrinks the imbalance that has to be corrected. The same per-image region feature can then do two more jobs. It can shift the classifier toward this image's appearance of the class, and it can be the unit over which an overlap loss (Dice, defined below) is computed, so that a 30-pixel rare region counts as much as a 30,000-pixel road. The per-pixel loss stays responsible for keeping probabilities calibrated, because pseudo-label selection depends on calibrated probabilities.

Core Claim

In semi-supervised, long-tailed segmentation, class rebalancing should act on soft region features computed from teacher probabilities, not on pixels. Calibration and rebalancing should be handled by separate loss terms: per-pixel cross-entropy keeps the probabilities usable for pseudo-label selection, and region-level terms (a frequency-balanced region classifier plus a per-region Dice term) carry the rebalancing. We predict this will raise rare-class mIoU without the false-positive inflation of pixel reweighting, and without the calibration damage that Dice-heavy training does to pseudo-labeling. Instance-conditioned classifier adaptation is a secondary component. It is kept only if a gated version beats a static classifier on tail classes when teacher masks are noisy.

Grounding in Prior Work

Two results motivate the region unit. Region Rebalance for Long-Tailed Semantic Segmentation shows that pixel-level loss reweighting raises false positives and lowers mIoU. It replaces pixel reweighting with a "Gather" step that averages each class's pixels into one region feature, trained in an auxiliary branch with Balanced Softmax, a cross-entropy that shifts each logit by the log of the class's prior frequency. The paper reports that the imbalance factor on COCO-Stuff164K drops from 2612 for pixels to 528 for regions. Its Gather needs ground-truth masks, and the authors leave predicted or semi-supervised regions to future work. That open problem is the one we take up.

On the semi-supervised side, Adaptive Equalization Learning tracks per-class confidence and uses it to drive copy-paste, CutMix, sampling and pixel reweighting. DARS replaces a single confidence threshold with per-class thresholds that make pseudo-label class proportions match the labeled split. Both rebalance at the pixel or patch level, which is the level Region Rebalance argues is wrong for mIoU.

The instance-adaptation component borrows its method from SSA-Seg. SSA-Seg's semantic prototype adaptation averages features under a coarse predicted mask and moves the fixed classifier weights toward this per-image center. Our soft region feature is the same masked average, so one computation serves both SSA-Seg's adaptation and Region Rebalance's gather. SSA-Seg was tested only with full supervision, and its own record warns that a corrupt coarse mask makes adaptation reinforce the error. That warning constrains our design and is why the adaptation is gated. Similar work in zero-shot learning, which shifts class attribute vectors per image (Dual Feature Augmentation Network, PSVMA), supports the general idea but not this setting.

Prototype methods for semi-supervised segmentation, namely Prototype-based Consistency Regularization and BRPG, keep dataset-wide running prototypes rather than per-image ones. They do not target the tail. The first explicitly notes that it is beaten by AEL on Cityscapes 1/16, which it attributes to AEL's imbalance handling.

On the loss side, Segmentation Loss Odyssey supplies the taxonomy of loss types and the SegLoss implementations. It also notes that region losses yield overconfident probabilities and have unstable gradients when predicted overlap is near zero. Compound cross-entropy-plus-Dice objectives with fixed weights, as in the white matter hyperintensity work and Enhanced U-Net, handle binary foreground-background imbalance only. We use Dice differently: as a per-region overlap term restricted to regions with enough soft mass, not as a global replacement for cross-entropy.

Research Program

The base system is a standard weak/strong-augmentation teacher-student pipeline. This is the setting in which PrevMatch reports gains over UniMatch and AugSeg. On each unlabeled image, the teacher, seeing a weakly augmented copy, produces class probabilities $q_{ic}$ for pixel $i$ and class $c$. The student sees a strongly augmented copy aligned to the teacher's view and produces features $f_i$. The soft region feature and its mass are

$$r_c = \frac{\sum_i q_{ic}\, f_i}{\sum_i q_{ic}}, \qquad m_c = \sum_i q_{ic},$$

and a region is kept only if $m_c$ exceeds a small floor of a few pixels' worth of probability. A rare-class pixel the teacher gives probability 0.3 therefore still contributes to the rare-class region, whereas a 0.95 threshold would discard it. On labeled images $q_{ic}$ is the one-hot ground truth, which recovers Region Rebalance's Gather exactly.

Three objectives act on these quantities. The first is per-pixel cross-entropy on labeled pixels and on confident pseudo-labeled pixels. It is the only term that shapes the probabilities used to select pseudo-labels. The second is a region classifier trained with Balanced Softmax on each $r_c$, with the class prior $\pi_c$ estimated online as the fraction of kept soft regions belonging to class $c$. Because it counts images containing a class rather than pixels, this prior is less extreme and less noisy than the pixel counts that Seesaw-style losses need. The third is a soft Dice term computed separately for each (image, class) region between student and teacher masks and averaged over regions. This makes overlap size-invariant at the region level, and the mass floor keeps it away from the zero-overlap regime where its gradients vanish.

Instance adaptation then sets the per-image classifier weight

$$w_c(x) = w_c + \lambda_c(x)\,\big(P r_c - w_c\big),$$

where $w_c$ is the static weight, $P$ a learned projection, and $\lambda_c(x)\in[0,1]$ a gate. The gate is computed from region mass and from disagreement between the current teacher and an ensemble of past checkpoints in the style of PrevMatch. It closes when the teacher's region is unstable, which is exactly the drift case SSA-Seg warns about.

The analysis tracks four quantities throughout training on each split. They are tail-class pseudo-label recall and precision, the per-class false-positive area, the expected calibration error of the probabilities used to select pseudo-labels, and the distribution of gate values per class.

Baselines

The naive reference is the plain teacher-student pipeline with a fixed confidence threshold and pixel cross-entropy, trained on the same splits, backbone and schedule. Every gain is reported relative to it.

The strongest existing competitor on this exact question is AEL. It is the published method that targets long-tailed semi-supervised segmentation directly, and it reports 74.28% mIoU on Cityscapes with 1/32 labels and 76.97% mIoU on PASCAL VOC with 1/32 labels (AEL). We will rerun AEL from its description on our backbone and schedule, giving it the same hyperparameter search budget as our method, and also quote its published numbers. Beating it means higher overall mIoU together with a clear gain on the rarest third of classes, averaged over three seeds and outside seed variance.

DARS is run the same way as the threshold-redistribution competitor. Prototype-based Consistency Regularization (73.41% mIoU on Cityscapes 1/16) and BRPG (73.62% on VOC with 92 labels) represent global prototypes. PrevMatch represents a strong recent pipeline improvement (+1.8% over UniMatch with 92 VOC labels). Our components are orthogonal to PrevMatch, so we will also report ours added on top of it.

Two constructed baselines isolate the key ingredient. The first is Region Rebalance with hard pseudo-label regions: Gather over thresholded pseudo-labels, which is the obvious extension its authors left open. The second is the same system with soft regions. The difference between them measures what soft gathering buys. A third ablation adds fixed global cross-entropy plus Dice with weights of one, to separate per-region Dice from ordinary compound loss. Finally, the gate is ablated by comparing $\lambda=0$ (no adaptation), $\lambda=1$ (ungated, SSA-Seg-style), and the learned gate.

For the long-tailed scarce setting, no reported result was found on Lacuna for semi-supervised segmentation on ADE20K-Full or COCO-Stuff-LT. There we compare against our own runs of AEL, DARS and hard-region Region Rebalance. As a fully supervised reference point we quote the Frequency-based Matcher's ADE20K-Full gain of 1.5 mIoU overall and 3.5 on rare classes (Frequency-based Matcher).

Why not just…?

Why not just add class-balanced cross-entropy or focal loss to a standard semi-supervised pipeline? Region Rebalance reports that pixel-level reweighting raises false positives and lowers mIoU, and AEL argues that frequency is a poor proxy for difficulty. We will still run class-balanced cross-entropy and focal loss inside our pipeline and report per-class false-positive area next to mIoU. Our claim is falsified if pixel reweighting matches region rebalancing on rare-class mIoU without the false-positive increase.

Why not just run Region Rebalance on thresholded pseudo-labels? This is the most serious threat, because it is simple and directly addresses the gap the Region Rebalance authors named. Our argument is that thresholding removes exactly the uncertain rare-class pixels whose regions matter most. The hard-versus-soft gather ablation settles it. If hard regions match soft regions on the rarest classes, the soft gather is not the contribution, and the note's claim reduces to "Region Rebalance extends to semi-supervised learning as-is." That is still useful, but it is a weaker paper.

Why not just use global multi-prototypes per class? Prototype-based Consistency Regularization and BRPG do this, but tail classes have too few samples to support several stable clusters, and SSA-Seg reports +4.17 mIoU for OCRNet on ADE20K from per-image adaptation over a fixed classifier. Those gains were obtained with full supervision, though, so they do not transfer automatically. The comparison of $\lambda=0$, ungated $\lambda=1$ and the learned gate, broken out by frequency band, decides whether adaptation survives noisy teacher masks. If it does not, it is dropped.

Why not just add a fixed cross-entropy plus Dice compound loss? A single Dice term pooled over all classes lets large classes dominate, and its gradients vanish for rare classes with near-zero overlap early in training. Dice-heavy training also makes probabilities overconfident, which undermines threshold-based pseudo-label selection. We test this by comparing the global compound loss against per-region Dice, and by measuring calibration error on the probabilities that select pseudo-labels.

Why not just use AEL-style adaptive copy-paste? It acts on the data and is compatible with our approach rather than a substitute for it. We report AEL alone, ours alone, and the two combined. If ours adds nothing on top of AEL, the region unit is redundant with confidence-guided augmentation.

Proposed Experiments and Decisive Checks

The main experiments use Cityscapes splits of 1/32, 1/16 and 1/8 labels, PASCAL VOC with 92 labeled images, and ADE20K with partial labels. A harder long-tail test uses ADE20K-Full and COCO-Stuff-LT with 1/16 and 1/8 of the images labeled. We report mIoU overall and by frequent, common and rare bands, with three seeds per configuration.

Four checks decide the claim. First, soft region gathering must beat hard pseudo-label gathering on rare-class mIoU, and the difference must grow as the label fraction shrinks. Second, region rebalancing must raise rare-class recall while keeping per-class false-positive area at or below the naive baseline, where pixel reweighting is expected to inflate it. Third, adding per-region Dice must not raise the expected calibration error of the probabilities used for pseudo-labels. Alongside this we report a variant in which Dice gradients also reach the pixel head, to show whether keeping the two roles separate actually matters. Fourth, in a stress test where a fraction of teacher masks are randomly reassigned to head classes, the gated adaptation must degrade more gracefully than the ungated version. If the gate cannot beat $\lambda=0$ on tail classes in this test, instance adaptation is removed from the method.