Introduction
State-of-the-art instance segmentation models, such as Mask R-CNN, achieve impressive results on balanced datasets like COCO. However, their performance craters when applied to real-world "long-tail" datasets where a few categories are abundant while hundreds of others are rare. This paper investigates the root cause of this failure and demonstrates that the primary bottleneck is not in object localization or mask generation, but specifically within the classification head.
The Classification Bottleneck
In long-tail scenarios, standard training procedures lead to a severe classification bias toward frequent "head" classes. To diagnose this, the authors compared performance on the balanced COCO dataset versus the long-tailed LVIS dataset. While the Region Proposal Network (RPN) remains surprisingly robust—successfully finding rare objects even with minimal training examples—the model consistently fails to label them correctly.
As shown in the empirical analysis above, replacing predicted labels with ground-truth labels for generated proposals causes the Mask AP on rare classes to jump from near-zero to over 40%. This confirms that the model's ability to localize and segment objects is already quite capable; the "devil" lies in the final classification step.
The SimCal Framework
To address this, the authors propose SimCal (Simple Calibration), a framework that decouples feature learning from classifier calibration. The approach consists of two main stages:
- Decoupled Calibration with Bi-level Sampling: Instead of balancing the entire training process, which can degrade feature learning for common classes, the authors first train a standard model. They then freeze the backbone, RPN, and mask heads, retraining only the classification head. This is done using a Bi-level Sampling strategy: first sampling categories uniformly, and then sampling images/instances within those categories. This ensures the classifier sees a balanced distribution of features without overfitting the entire network to rare class images.
- Dual Head Inference: To maintain high performance on frequent classes while boosting rare ones, SimCal employs a dual-head approach during inference. It uses the calibrated head for rare classes and the original, biased head for frequent classes.
Key Results
SimCal provides a significant boost to long-tail performance without requiring complex architectural changes. On the LVIS v0.5 benchmark, applying SimCal to a ResNet-50 Mask R-CNN improved the overall Mask AP from 18.0 to 23.4. The most dramatic gains were seen in rare classes ($AP_r$), which increased from 2.7 to 16.4.
When integrated with more advanced models like the Hybrid Task Cascade (HTC), the framework achieved 33.4 AP, setting a new state-of-the-art at the time. This simple yet effective calibration strategy was the core of the winning entry for the 2019 LVIS Challenge.
Conclusion
By identifying classification bias as the central challenge in long-tail instance segmentation, this work shifts the focus from complex architectural modifications to simple, decoupled calibration. SimCal demonstrates that even with very few training instances, models can learn to recognize rare objects effectively if the classification head is properly recalibrated.
Summary updated