Some instance segmentation methods do not draw boxes around objects. Instead, a network gives every pixel a vector, and pixels of the same object are trained to have nearby vectors while different objects are pushed apart. Objects are then recovered by clustering the vectors. These discriminative instance embeddings handle cases that boxes handle badly, such as touching nuclei, overlapping leaves, winding lanes and catheters. They have almost never been studied when object categories are heavily imbalanced, where a few classes supply most examples and many classes are rare. That is the normal condition in driving scenes, indoor 3D scans and large-vocabulary datasets.
We propose to find where the damage from imbalance lands in these models before trying to repair it. The embedding loss ignores categories and averages over instances, so we expect it to group rare objects about as well as common ones. We expect the failure to sit almost entirely in the separate per-pixel semantic branch that names each cluster. If a controlled oracle study confirms this, the fix is to stop labeling pixels and instead label whole clusters. That cluster classifier is trained in a second, class-balanced stage while the grouping network stays frozen. If the study refutes it, we learn that the pull/push geometry itself is frequency-biased, and the proposal switches to a clearly defined alternative repair.
Core Claim
The discriminative loss of Semantic Instance Segmentation with a Discriminative Loss Function has three parts. One term pulls each pixel toward its instance's mean vector until it is within a margin $\delta_v$. A second term pushes instance means apart until they are $2\delta_d$ apart. A third term keeps the means near the origin. Each term is averaged over the $C$ instances in the image, so a rare-class instance present in an image gets the same weight as a frequent one. Frequency enters only indirectly, through how often an instance type appears and which other types it appears alongside.
The semantic branch that names clusters is different. In Semantic Instance Segmentation for Autonomous Driving and in the joint point-cloud model JSNet, this branch is ordinary per-pixel cross-entropy. That loss is weighted twice over toward common classes: once by how often each class occurs, and again by how many pixels it covers. De Brabandere et al. already note that errors in this branch pass straight through to the final masks, and that clustering cannot recover a missed class.
Our claim has two parts. First, under long-tailed data, the bottom-up pipeline's loss on rare classes comes mostly from labeling, not grouping. Second, the right intervention is cluster-level classification recalibrated in a decoupled stage. Frequency-aware reweighting of the pull/push terms is the wrong intervention.
This is the bottom-up counterpart of the diagnosis in The Devil Is in Classification. That paper showed that in Mask R-CNN on LVIS, swapping predicted labels for ground-truth labels raises rare-class mask AP from near zero to over 40%, which places the fault in the box classifier. That finding cannot be assumed to carry over. Bottom-up models have no proposal network and no separate classification head. Their grouping is learned jointly with pixel semantics, and they are known to degrade on small instances, which could interact with rarity.
What Prior Work Establishes
Three papers supply the imbalance mechanisms we borrow or contrast against, and all three sit inside box-based detectors.
- SimCal (The Devil Is in Classification) freezes the backbone and mask head. It then retrains only the classifier with bi-level sampling: pick classes uniformly, then pick instances within the class. We reuse this decoupling, but apply it to classifiers over pooled cluster features rather than box proposals.
- Equalization Loss (Equalization Loss for Large Vocabulary Instance Segmentation) zeroes the negative gradient that frequent-class proposals send to rare classes.
- DropLoss (DropLoss for Long-Tail Instance Segmentation) shows that background supplies 50–70% of the discouraging gradients for rare classes versus 30–40% for frequent ones. It also shows that fixed reweighting trades frequent-class AP for rare-class AP. This motivates our design: per-pixel semantic heads are dominated by background and large-area pixels, even more than box classifiers are. It also constrains our design: any reweighting we test must be judged on the frequent-class side too.
The bottom-up works supply the grouping mechanism, and each assumes balance in some way:
- Weakly Supervised Nuclei Segmentation via Instance Learning uses a single "nucleus" class.
- JSNet trains its semantic head with plain cross-entropy and reports only aggregate precision and IoU on S3DIS.
- Towards End-to-End Lane Detection has essentially one foreground class.
Two metric-learning methods for long-tailed data, ProCo and CCL-Net, learn class prototypes for whole-image classification. Their embedding pulls all members of a class together, which directly conflicts with an instance embedding that must push two cars apart. We therefore keep the instance geometry and the class decision in separate spaces rather than merging them. We use a prototype classifier only as one variant of the cluster classifier.
Research Program
Stage one: the decomposition study. We train a standard discriminative-embedding model with an ordinary per-pixel semantic head on long-tailed data. We then evaluate it in a two-by-two grid:
- predicted clusters with predicted labels;
- predicted clusters with oracle labels, where each predicted cluster takes the majority ground-truth class of its pixels;
- oracle grouping with predicted labels, where ground-truth instance masks are named by the model's semantic branch;
- the full oracle, as a ceiling.
Rare classes are often small, and small instances are a known weakness of the embedding. Every comparison is therefore also stratified by instance area, so that rarity is not confused with size. We also measure the embedding geometry directly for each pair of classes:
- how far instances of that pair sit from each other in embedding space;
- how often they appear in the same training images.
This tests whether rare classes that seldom co-occur with others receive too little push. The hypothesis predicts two things. Under oracle labels, the rare-versus-frequent gap in AP should shrink to roughly what instance size alone explains. Under oracle grouping, the gap should stay large.
Stage two: the repair. The repair follows whichever cell of the grid carries the loss.
If labeling dominates, as predicted, the architecture becomes:
- the embedding head;
- a foreground/background head that ignores class, since a two-way split is far less exposed to the long tail;
- a classifier over clusters.
The classifier takes backbone features averaged over each cluster's pixels. In a second stage, the backbone, embedding head and foreground head are frozen. The classifier is trained on features pooled over ground-truth instance masks from the training set, using SimCal-style bi-level sampling. At test time it is applied to the clusters found by mean shift, the clustering method that finds density peaks without being told the number of objects.
Three choices remove the pixel-area weighting and most of the background pressure:
- one training example per instance, not per pixel;
- balanced sampling;
- no background class in the classifier.
We compare three forms of the classifier: a linear head, a cosine-similarity head, and a nearest-prototype head in the spirit of ProCo.
If instead grouping dominates for rare classes, the repair is to rebalance the push term. We would add a push-only term that draws rare-class centroids from a memory bank of stored instance means, so rare instances are pushed against classes they rarely meet in the same image. Stage one decides which branch we build. We report both the prediction and the outcome.
Baselines
Naive reference. The unmodified discriminative-loss model with a per-pixel semantic head, trained on the natural distribution. On Cityscapes, Semantic Instance Segmentation with a Discriminative Loss Function reports 17.5 AP and 35.9 AP50. On S3DIS Area 5, JSNet reports 62.1% mean precision and 54.5% mean IoU. We re-run both ourselves with the same backbone and schedule, because neither paper reports per-class results split by frequency. No reported result was found on Lacuna for per-class or head-versus-tail breakdowns of either model.
Data rebalancing. The same model trained with Repeat Factor Sampling from LVIS. This resamples images by the square root of inverse class frequency, and in Mask R-CNN it raised rare-class AP from 0.8 to 13.4. It is the cheapest serious competitor. It applies to bottom-up models unchanged, and it helps the embedding and the semantic head at the same time.
Loss reweighting. The per-pixel semantic head trained with class-balanced weights from Class-Balanced Loss Based on Effective Number of Samples, with $\beta$ tuned on validation data. We also run an Equalization-Loss-style suppression of frequent-to-rare negative gradients in that head.
Strongest existing system. On LVIS we compare against box-based long-tail methods at matched ResNet-50 backbones. The Devil Is in Classification reports SimCal at 23.4 mask AP and 16.4 AP_r on LVIS v0.5. DropLoss for Long-Tail Instance Segmentation reports 25.5 AP and 13.2 AP_r, and 26.4 AP when combined with Repeat Factor Sampling.
Key ablation. This isolates cluster-level labeling from decoupling. We train the cluster classifier jointly with the rest of the network (no second stage), and separately apply the second-stage balanced retraining to the original per-pixel head.
Fairness and success criteria. All variants on a given dataset share the backbone, the input resolution, the total training iterations including the second stage, and a matched validation tuning budget of equal trials per method. On Cityscapes and S3DIS, beating the baselines means two things. The rare-class average precision, taken over the least frequent third of classes by training instance count, must improve over the best of Repeat Factor Sampling and class-balanced loss by more than the spread across three seeds. The frequent-class AP must also fall by no more than that spread. On LVIS we do not expect to beat SimCal overall, given the known weakness of bottom-up models on diverse COCO-style scenes. Success there means closing a large share of the rare-class gap between the naive bottom-up model and SimCal's 16.4 AP_r.
Why not just…?
Why not just use Repeat Factor Sampling? It is the most plausible competitor, and it may be enough on datasets with mild imbalance. The worry is that resampling images leaves the per-pixel head's area weighting and background dominance in place: an oversampled image of a rare bus is still mostly road and cars. The decomposition study measures this directly. If Repeat Factor Sampling alone closes the oracle-label gap, the cluster classifier is unnecessary, and we will report that. We also test the two combined, since they are not mutually exclusive.
Why not just reweight the pixel classifier with class-balanced or focal weights? Per-class weights correct for class counts but not for the pixel area within a class, which is a separate source of bias. DropLoss also found that fixed reweighting buys rare-class gains at a frequent-class cost. The class-balanced $\beta$ must also be retuned per dataset, and counting "samples" is ambiguous when the unit is a pixel. Our baseline arm uses a properly tuned version of this reweighting, and the frequent-class criterion above checks the trade-off.
Why not just use a box-based detector with SimCal? For LVIS-like scenes with scattered, compact objects, that is probably the better choice, and we say so. The reason to fix bottom-up models is the class of scenes where boxes fail: touching, overlapping and elongated instances. Examples are S3DIS furniture clutter and crowded Cityscapes scenes. We will compare against Mask R-CNN with SimCal on Cityscapes at matched backbone and schedule, reporting AP separately for occluded and non-occluded instances. The bottom-up approach earns its place only if it wins where occlusion is heavy.
Why not just add class-balanced weights to the pull/push loss? That would be the obvious first attempt at this intersection. Our hypothesis says it targets the wrong component. The oracle-label cell of the stage-one grid answers this directly, and we also train this variant as an extra arm.
Decisive Experiments
Datasets. Cityscapes has eight instance classes of very unequal frequency, which we will count from the training set. S3DIS Area 5 covers the 3D joint semantic-and-instance setting. LVIS v0.5 is a stress test, reported with AP_r, AP_c and AP_f. For Cityscapes and S3DIS we define head, middle and tail by training instance count, and report metrics per group.
Result that sets the direction. The stage-one grid, stratified by instance size.
- The labeling hypothesis is falsified if two things hold together. Under oracle labels, tail-class AP still trails head-class AP by clearly more than size-matched head instances would explain. And under oracle grouping, the tail gap is small.
- In that case the project turns to the memory-bank push term. The cluster-classifier results then become a negative result.
Result that validates the repair. On Cityscapes and S3DIS, the decoupled cluster classifier must beat Repeat Factor Sampling and class-balanced reweighting on tail-group AP with no significant head-group loss. The ablation must also attribute the gain to cluster-level labeling plus decoupling, rather than to extra training iterations.
Secondary check. Whether the class-pair centroid distances track how often the pair co-occurs in training images. A strong positive relationship would show that the push term carries a frequency bias, even if labeling dominates the overall error.
References
- Semantic Instance Segmentation with a Discriminative Loss Function
- Semantic Instance Segmentation for Autonomous Driving
- JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds
- The Devil Is in Classification: A Simple Framework for Long-Tail Instance Segmentation
- Equalization Loss for Large Vocabulary Instance Segmentation
- DropLoss for Long-Tail Instance Segmentation
- Weakly Supervised Nuclei Segmentation Via Instance Learning
- Towards End-to-End Lane Detection: an Instance Segmentation Approach
- ProCo: Prototype-Aware Contrastive Learning for Long-Tailed Medical Image Classification
- Learning and Exploiting Interclass Visual Correlations for Medical Image Classification
- LVIS: A Dataset for Large Vocabulary Instance Segmentation.
- Class-Balanced Loss Based on Effective Number of Samples.