Methods for long-tailed segmentation report large improvements on rare classes. Long-tailed here means a few common classes supply most of the training examples while hundreds of rare classes have only a handful. These methods train on top of backbones that were pre-trained on large image collections, and nobody has checked how much of the rare-class gain depends on the backbone having already seen those categories, or even the exact test images. We propose a controlled experiment that answers this. We pre-train the same backbone several times, varying exactly one thing: how much it has seen of the rare categories, from not at all, to the category but different images, to the evaluation images themselves. We then fine-tune each backbone with a set of long-tailed and hybrid segmentation losses and measure which reported gains persist as exposure is removed.
The point is to separate two kinds of memorization that current practice lumps together. Does Learning Require Memorization? A Short Tale about a Long Tail proves that memorizing rare examples is necessary for good accuracy on long tails. The contamination literature, by contrast, treats memorized evaluation items as an evaluation failure (Investigating Data Contamination in Modern Benchmarks for Large Language Models). Both views can be right. Seeing other images of a rare category is legitimate tail learning, while seeing the test image is contamination. A segmentation benchmark can tell these apart only if exposure is controlled. Our specific hypothesis is that losses which rebalance the classifier mostly resurface category knowledge the backbone already has, while region-overlap losses improve mask outlines in a way that does not depend on exposure.
Core Claim
Rare-class error in instance segmentation has two parts. The first is recognition: naming the object correctly. The second is delineation: drawing its mask well. We predict that pre-training exposure mainly helps recognition. Classifier-side rebalancing methods therefore should lose most of their rare-class gain when the backbone has never seen the category.
Seesaw Loss for Long-Tailed Instance Segmentation reweights the gradients that frequent classes impose on rare ones in the classification branch of Mask R-CNN. Our prediction applies to it, and to post-hoc classifier rescaling of the kind in Decoupling Representation and Classifier for Long-Tailed Recognition.
Hybrid losses add a region-overlap term to per-pixel cross-entropy. That term is Dice, which scores the overlap between predicted and true masks normalized by their sizes, so a tiny object counts as much as a large one. We predict hybrid losses improve delineation by a roughly constant amount whatever the exposure. Separately, we predict that injecting evaluation images into self-supervised pre-training inflates rare-class AP more than frequent-class AP, because rare classes have little fine-tuning data to override what was memorized.
If both predictions hold, two changes to reporting practice follow. Reported rare-class AP on these benchmarks should be split by exposure. Rebalancing papers should separate recognition gains from delineation gains. If the predictions fail, the field gets the opposite and equally useful result: published tail gains are robust to exposure.
Prior Work and Its Role
Three groups of work motivate this audit.
The contamination group shows that pre-training leakage inflates downstream scores. Both Text and Images Leaked! finds leakage in multimodal models inherited from unimodal pre-training. Work on implicit shot bias and category leakage in few-shot detection, such as Revisiting Few-Shot Object Detection with Vision-Language Models, shows that "novel" categories are often present in pre-training data. None of these studies look at dense segmentation or at how training losses interact with leakage.
The long-tailed segmentation group supplies the methods we audit. Seesaw Loss raises LVIS rare-class AP from 0.8% to 18.1% but rebalances only the classifier and keeps plain binary cross-entropy in the mask head. Region Rebalance for Long-Tailed Semantic Segmentation pools pixels of each class into region features and trains them with Balanced Softmax. It also constrains our design: it shows that naive pixel reweighting raises false positives and lowers mean intersection-over-union (mIoU). For that reason we do not propose a new reweighted loss.
The hybrid-loss group supplies the region term. Compound BCE-Dice losses for brain lesions (Deeply supervised network for white matter hyperintensities segmentation with transfer learning) and IoU-Focal-Dice losses for polyps (Enhanced U-Net) target foreground-versus-background imbalance in binary tasks. We move the Dice term into the per-instance mask head of a many-class long-tailed model, where each mask is again a binary problem but rare classes have few training masks.
We deliberately avoid the obvious combination of LDAM margins (Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss) with Dice in one loss. Without an exposure control, any gain from such a loss would carry the same confound we want to measure.
Research Program
The central artifact is an exposure ladder. We use a fixed ResNet-50 and a fixed self-supervised pre-training recipe on ImageNet-scale data, with only the pre-training data varied across five rungs:
- No pre-training. The backbone is trained from scratch, as a floor.
- Category removed. All images of categories matching the held-out set of LVIS rare categories are removed. Matching uses category names and synsets.
- Category present. The standard pre-training set with those categories included.
- Contamination, small. The standard set plus the LVIS validation images, unlabeled, so the backbone has seen the test pixels but never the masks or labels.
- Contamination, large. The same injection at larger dose: the validation images are also repeated, to model the heavy duplication typical of web data.
Rungs 2 and 3 isolate legitimate category exposure, the Feldman-type memorization. Rungs 3 to 5 isolate instance contamination. Self-supervised pre-training keeps labels out of the picture.
Each backbone is then fine-tuned on LVIS v1 with Mask R-CNN under the same schedule. We use six loss configurations:
- plain cross-entropy with a BCE mask head;
- Seesaw in the classifier;
- repeat-factor sampling plus Seesaw;
- a hybrid BCE-Dice mask head with a plain classifier;
- Seesaw combined with a hybrid BCE-Dice mask head;
- decoupled classifier retraining in the style of Kang et al.
We repeat a smaller version on ADE20K semantic segmentation with DeepLabv3+, comparing cross-entropy, Region Rebalance, and cross-entropy plus Dice. Rare classes there are defined by pixel frequency.
To separate recognition from delineation, we report rare, common, and frequent mask AP, each in three forms. The first is standard. The second uses an oracle that replaces predicted class labels with ground-truth labels for matched detections, which measures delineation alone. The third compares box AP to mask AP, which measures recognition and localization without mask quality. Our quantity of interest is the exposure sensitivity of each loss's gain: how much the gain over the cross-entropy baseline changes between rung 2 and rung 3, and between rung 3 and rungs 4 and 5. We use three seeds per cell. Pre-training is the expensive step, about five backbones in total. Fine-tuning runs are numerous but standard.
Baselines
The naive reference is Mask R-CNN with cross-entropy and a BCE mask head, fine-tuned from each rung. All gains are measured against it at the same rung. This guards against misreading a better backbone as a better loss. For context, LVIS reports Mask R-CNN falling from 36.4% to 9.8% mask AP when trained on a 1k-image subset. That shows how fragile the low-data regime is.
The strongest existing classifier-side method is Seesaw Loss, especially with repeat-factor sampling and a Cascade Mask R-CNN, where its paper reports 30.1% AP on LVIS v1. It also reports rare-class AP rising from 0.8% to 18.1% with standard Mask R-CNN. We rerun it with the authors' hyperparameters on our ResNet-50 backbones, so absolute numbers will be lower. We will first check that, at rung 3, our Seesaw rare-class gain over cross-entropy is within a few points of the reported gain. Only then do we interpret the other rungs.
The strongest semantic-segmentation competitor is Region Rebalance. It reports +1.61% mIoU for DeepLabv3+ ResNet-101 on ADE20K and +6.44% for OCRNet on COCO-Stuff164K. We rerun it on DeepLabv3+ with the same schedule at every rung.
For the hybrid BCE-Dice mask head inside a long-tailed instance segmenter, there is no reported result on Lacuna for LVIS. Its compound-loss sources report only binary medical results, such as 82.3% Dice on the WMH challenge data. We tune its Dice weight on a held-out split of LVIS training images, with the same tuning budget as Seesaw: five configurations each. Decoupled classifier retraining is the post-hoc baseline.
The key ablation is Seesaw with and without the Dice mask term. It tests whether classifier and mask interventions add up, and whether they depend on exposure differently.
"Beating" in this proposal does not mean a higher absolute AP. It means a finding that holds across seeds. For one, we would show that some loss's rare-class gain shrinks by more than half from rung 3 to rung 2 while another's does not. For another, we would show contamination inflating rare-class AP beyond its effect on frequent-class AP, by more than the variation across seeds.
Why not just…?
Why not just deduplicate LVIS against the pre-training set and report clean numbers? Near-duplicate filtering corresponds to rungs 3 versus 4. It catches instance contamination but says nothing about category exposure, which we expect to be the larger effect for rare classes. Retrieval and n-gram methods in language also miss paraphrased leakage (Investigating Data Contamination in Modern Benchmarks for Large Language Models), and the visual analogue is probably similar. Deduplication also cannot tell us which losses depend on exposure.
Why not just stratify rare classes by how often their names appear in existing pre-training data, with no new pre-training? This is the real threat, because it is far cheaper. Term frequency does predict few-shot performance in language models (Snoopy). The problem is confounding: categories that are common on the web may also be visually easier or larger. We will run this observational analysis on public backbones as well. If its exposure-sensitivity estimates match the controlled ladder at rungs 2 and 3, we will recommend it as the cheap protocol. The ladder is what validates it.
Why not just train from scratch and avoid exposure entirely? Rung 1 does exactly this. But scratch-trained backbones are so weak on LVIS that rare-class AP sits near zero for every loss, and that floor hides any difference between losses. The informative contrast is between rungs that differ only in rare-category content.
Why not just design a better loss, such as LDAM margins added to Dice? Any new loss evaluated on a pre-trained backbone carries the confound this study measures. Building such a loss is a good follow-up once we know which kind of gain depends on exposure.
Decisive Checks
Three results would settle the hypothesis.
- Seesaw versus the hybrid term under category removal. If Seesaw's rare-class gain largely disappears at rung 2 while its oracle-class (delineation) gain stays, and the hybrid Dice term's gain is flat across rungs 2 and 3, the recognition-versus-delineation account holds. If both shrink equally, exposure helps everything and the decomposition is wrong.
- Contamination inflation by class frequency. Rungs 4 and 5 must raise rare-class AP more than frequent-class AP for the contamination concern to matter in dense prediction. A null result here would be reassuring and still worth publishing.
- Transfer to ADE20K. The same pattern must appear on ADE20K for Region Rebalance versus the Dice term. If it holds only on LVIS, the effect is tied to instance-level recognition rather than to long tails in general.
References
- Does learning require memorization? a short tale about a long tail
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- Seesaw Loss for Long-Tailed Instance Segmentation
- Decoupling Representation and Classifier for Long-Tailed Recognition
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- Revisiting Few-Shot Object Detection with Vision-Language Models
- Region Rebalance for Long-Tailed Semantic Segmentation
- Deeply supervised network for white matter hyperintensities segmentation with transfer learning
- Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation
- Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- LVIS: A Dataset for Large Vocabulary Instance Segmentation.
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance