Imprinted Centroids as the Missing Half of Hybrid Losses for Rare Segmentation Classes

Semantic segmentation assigns a class to every pixel. In real scene datasets a few classes, such as wall, sky and road, cover most pixels, while hundreds of rare classes cover almost none. Two common fixes for this imbalance both have known flaws. The first is to reweight the per-pixel cross-entropy loss. This makes the model predict rare classes more often, but it floods the image with false positives, and mean intersection-over-union (mIoU, the standard segmentation score) goes down. The second is to add a Dice loss, an overlap score computed per class and normalized by that class's area, so a 10-pixel object counts as much as a 10,000-pixel one. Dice penalizes false positives by construction, which makes it a natural fit for mIoU. In practice, though, it is unstable when a class's prediction barely overlaps its target, and it produces overconfident probabilities. Rare classes are exactly the ones whose predictions barely overlap early in training.

This proposal argues that Dice fails on rare classes because of how their classifier weights are formed, not because of the loss itself. We replace the learned classifier with weight imprinting: each class's weight vector is set directly to the normalized average feature of that class's pixels, pooled under the ground-truth mask. Two things follow. First, a rare class's predictions land on its own pixels from the first update, so the overlap term has something to refine. Second, fixed-length imprinted weights paired with normalized features cap how large logits can grow, which removes one route to Dice's overconfidence. The claim is an interaction: an imprinted head together with a class-averaged Dice term should raise rare-class IoU and calibration by more than either component gives alone.

Core Claim and Its Position in Prior Work

Four strands of prior work motivate the claim.

Imprinting as a technique comes from Low-Shot Learning With Imprinted Weights, which writes mean normalized embeddings into the classifier without backpropagation. Adaptive Masked Weight Imprinting for Few-Shot Segmentation extends this to dense prediction with masked average pooling into a 1×1 convolutional head. It works only in 1- and 5-shot binary episodes, trains with plain pixel classification loss, and never meets a long tail. We reuse its pooling operator but apply it to all classes of a fully supervised, long-tailed, many-class problem.

The theoretical reason to expect imprinting to help under imbalance comes from Neural Collapse by Design: Learning Class Prototypes on the Hypersphere. It shows three things: on the unit sphere, class feature means are the optimal classifier weights; learned heads keep a "radial" freedom to lower loss by inflating vector lengths; and fixed-prototype heads help under severe imbalance, by up to +8.7%. That work is whole-image classification only and uses no overlap loss. Seesaw Loss for Long-Tailed Instance Segmentation supplies the matching observation in dense prediction: head-class weight norms grow far larger than tail norms. Seesaw fixes this with normalized activations plus frequency-driven gradient mitigation, but keeps learned weights.

The diagnosis we build on is Region Rebalance for Long-Tailed Semantic Segmentation. It shows that pixel reweighting trades false negatives for many more false positives. It also shows that imbalance is far milder at the region level, with a ratio of 528 against 2,612 for pixels on COCO-Stuff164K. Its fix is an auxiliary branch that averages each class's pixels into a region feature and trains that branch with Balanced Softmax. Our imprinted weight is exactly that region feature, used as the classifier itself rather than as an auxiliary training signal. Our overlap loss also attacks the false-positive problem directly, where Region Rebalance only does so indirectly.

The closest existing system is Rethinking Semantic Segmentation: A Prototype View (ProtoSeg). It already replaces the learned head with momentum-updated class prototypes on ADE20K-Full. It trains with cross-entropy plus pixel-to-prototype contrastive and distance terms, has no overlap loss, and reports no head/tail analysis. We differ from ProtoSeg in the loss we test, in the question we ask, and in the evaluation we run. We do not claim that prototype heads are new. If ProtoSeg alone captures the tail gains, that is a publishable negative finding, and our design detects it.

Two papers constrain the design. Unified Focal loss finds that symmetric Dice/focal hybrids fail under imbalance unless background suppression is tuned asymmetrically, so we must include its asymmetric hybrid as a baseline. Class Adaptive Network Calibration shows that rare classes are the most miscalibrated. This makes calibration a primary outcome for us, not an afterthought.

Mechanism

Let $\hat f_i$ be the unit-normalized decoder feature at pixel $i$. Let $\hat\mu_c$ be the unit-normalized running mean of features at pixels labeled $c$, accumulated over training batches by masked average pooling. Let $s$ be a fixed temperature. The imprinted head computes logits

$$z_{ic} = s\,\langle \hat f_i, \hat\mu_c\rangle ,$$

so every logit lies in $[-s, s]$ and no class can win by growing its weight norm. Training minimizes

$$L = L_{\text{CE}} + \lambda\,\frac{1}{|\mathcal C_B|}\sum_{c\in\mathcal C_B}\Big(1-\frac{2\sum_i p_{ic}y_{ic}}{\sum_i p_{ic}+\sum_i y_{ic}}\Big).$$

Here $p_{ic}$ is the softmax probability of class $c$ at pixel $i$, $y_{ic}$ is the one-hot label, and $\mathcal C_B$ is the set of classes present in the batch. The Dice term is computed over the whole batch rather than per image, so that rare classes have enough pixels for a stable ratio, and $\lambda$ weights it against cross-entropy. Gradients flow into the encoder only; the prototypes are updated by the running mean, not by backpropagation.

Rare classes appear in few batches. Their running mean therefore uses a count-based average, with a fixed floor on the weight given to each new appearance, instead of a single exponential-moving-average constant. Nothing in the method needs dataset-wide class frequencies, unlike Seesaw, Balancing Logit Variation and Balanced Softmax.

The mechanistic prediction is concrete. With a randomly initialized tail weight, the tail class's probability is spread over the image without regard to its location, so the Dice gradient mostly says "predict less of this class everywhere." With an imprinted weight, the probability is already concentrated on the class's region, so the same gradient sharpens boundaries and removes nearby false positives.

Research Program

We build the imprinted head as a drop-in replacement for the final 1×1 convolution in DeepLabv3+ with a ResNet-101 backbone and in a Swin-based UPerNet. The core study is a factorial design with two factors. The head is one of three types: a learned linear head, a learned cosine head with the same temperature, or the imprinted head. The loss is one of three: cross-entropy alone, cross-entropy plus class-averaged Dice, or the asymmetric Unified Focal hybrid. The learned cosine head is the ablation that matters most. It separates the effect of bounding logit lengths from the effect of setting weight directions to data centroids.

Beyond final scores, we log three quantities during training for each frequency group:

  • the cosine similarity between each class's Dice gradient on its probability map and its ground-truth mask;
  • the maximum logit value and weight norm for each class;
  • per-class false-positive and false-negative pixel rates.

Baselines

The naive reference point is the same network trained with plain pixel cross-entropy and a learned linear head. The file reports ~0.42 mIoU for PSPNet/UPerNet-class models on SceneParse150 (Semantic Understanding of Scenes through the ADE20K Dataset) and 33.2% mIoU for DeepLab V2 on COCO-Stuff (COCO-Stuff). We rerun this baseline rather than quote those figures, because backbones differ.

The strongest existing long-tail method is Region Rebalance. The file reports only relative gains for it: +1.61% mIoU over DeepLabv3+ ResNet-101 and +1.22% over Swin on ADE20K, and +6.44% over OCRNet on COCO-Stuff164K. No absolute value was found on Lacuna, so we reimplement it on our backbones and check that we reproduce roughly these deltas.

The strongest prototype competitor is ProtoSeg. It reports +1.2% mIoU on FCN and +0.8% on SegFormer on ADE20K, and +1.7% on ADE20K-Full. We run it with its own tripartite loss, with and without our Dice term.

We also include three further baselines:

  • Balancing Logit Variation (Balancing Logit Variation for Long-tailed Semantic Segmentation). It reports tail mIoU gains of 1.2–3.2% on Cityscapes, and we run it on Cityscapes as a secondary check.
  • The asymmetric Unified Focal hybrid on a learned head. This is the strongest hybrid loss in the file.
  • Post-hoc imprinting. We train the naive baseline, then replace its head with mask-pooled centroids at the end of training. This isolates whether imprinting must happen during training, while the encoder adapts to the overlap loss.

Fairness rules apply to every method: identical data, iterations, crop size and backbone initialization, and the same budget of hyperparameter trials. That budget is twelve configurations searched on a held-out split of the training set, covering $\lambda$ and $s$ for us, $\sigma$ for Balancing Logit Variation, the branch weight for Region Rebalance, and asymmetry weights for Unified Focal. We report three seeds.

Beating Region Rebalance means two things. The first is a higher tail-group mIoU than its reimplementation, by more than the seed standard deviation, with overall mIoU no lower. The second is that the tail-group false-positive rate is no higher than the naive baseline's.

Why not just…?

Why not just reweight the pixel cross-entropy by inverse frequency? Region Rebalance already measured this and found that mIoU falls, because false positives rise faster than false negatives fall. We include it at the naive tier only to confirm that finding on our backbones. It is not a serious threat.

Why not just add Dice to a standard learned head, as medical segmentation does? This is the real threat. If a learned head with cross-entropy plus Dice, or with Unified Focal, matches the imprinted head plus Dice on tail IoU, then the central claim is wrong. The factorial design tests exactly this through the head-by-loss interaction. The gradient-alignment and logit-norm logs show why the result comes out either way. We commit to reporting the interaction size whatever its sign.

Why not just use a learned cosine classifier, which already removes norm disparity as Seesaw's normalized activation does? Then the gain would come from bounded logits, not from centroids, and imprinting would be unnecessary. The learned cosine arm answers this. Our prediction is that it fixes calibration partly but not the early lack of overlap for tail classes, so its tail IoU under Dice should sit between the linear and imprinted heads.

Why not just use ProtoSeg or Region Rebalance as they are? Both are already strong, and ProtoSeg is structurally close to our head. ProtoSeg has no overlap term and was never evaluated by frequency group. Region Rebalance keeps a learned head. We run both unmodified and with the Dice term added. If ProtoSeg with Dice equals our single-centroid head, the contribution becomes the overlap-loss finding rather than the head design, and we will say so.

Proposed Experiments and Decisive Checks

The main benchmarks are ADE20K-150, ADE20K-Full with 847 classes (the longest tail in the file), and COCO-Stuff164K. Classes are split into head, body and tail by pixel-frequency terciles. The metrics are per-group mIoU, per-group false-positive and false-negative rates, and per-group expected calibration error (ECE, the gap between predicted confidence and actual accuracy).

Four outcomes decide the proposal:

  1. Interaction. The Dice term's tail-IoU gain must be larger under the imprinted head than under both learned heads.
  2. Where the gain comes from. The tail gain must come from fewer false negatives without more false positives, the opposite of what Region Rebalance documents for pixel reweighting.
  3. Calibration. Tail-class ECE under imprinted head plus Dice must be lower than under learned head plus Dice. Maximum logits must stay bounded while learned-head norms grow.
  4. Mechanism. In the first few thousand iterations, the alignment between the Dice gradient and the tail mask must be clearly positive for imprinted heads and near zero for learned ones.

If checks 1 and 4 fail, the mechanism is wrong. If only check 3 holds, the contribution narrows to a calibration fix for Dice.