The Devil Is in Classification: A Simple Framework for Long-Tail Instance Segmentation

Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Jun Hao Liew, Sheng Tang, Steven C. H. Hoi, Jiashi Feng

Introduction

State-of-the-art instance segmentation models, such as Mask R-CNN, achieve impressive results on balanced datasets like COCO. However, their performance craters when applied to real-world "long-tail" datasets where a few categories are abundant while hundreds of others are rare. This paper investigates the root cause of this failure and demonstrates that the primary bottleneck is not in object localization or mask generation, but specifically within the classification head.

The Classification Bottleneck

In long-tail scenarios, standard training procedures lead to a severe classification bias toward frequent "head" classes. To diagnose this, the authors compared performance on the balanced COCO dataset versus the long-tailed LVIS dataset. While the Region Proposal Network (RPN) remains surprisingly robust—successfully finding rare objects even with minimal training examples—the model consistently fails to label them correctly.

Figure (a) shows the severe long-tail distribution of LVIS; (b) compares performance on COCO vs LVIS; (c) demonstrates that using ground-truth labels (props-gt) for generated proposals dramatically improves performance, identifying classification as the primary culprit.

As shown in the empirical analysis above, replacing predicted labels with ground-truth labels for generated proposals causes the Mask AP on rare classes to jump from near-zero to over 40%. This confirms that the model's ability to localize and segment objects is already quite capable; the "devil" lies in the final classification step.

The SimCal Framework

To address this, the authors propose SimCal (Simple Calibration), a framework that decouples feature learning from classifier calibration. The approach consists of two main stages:

  1. Decoupled Calibration with Bi-level Sampling: Instead of balancing the entire training process, which can degrade feature learning for common classes, the authors first train a standard model. They then freeze the backbone, RPN, and mask heads, retraining only the classification head. This is done using a Bi-level Sampling strategy: first sampling categories uniformly, and then sampling images/instances within those categories. This ensures the classifier sees a balanced distribution of features without overfitting the entire network to rare class images.
  2. Dual Head Inference: To maintain high performance on frequent classes while boosting rare ones, SimCal employs a dual-head approach during inference. It uses the calibrated head for rare classes and the original, biased head for frequent classes.
The SimCal framework utilizes bi-level sampling (left) to provide balanced training data for the classification head and explores various head architectures (right), including the final dual-head inference model.

Key Results

SimCal provides a significant boost to long-tail performance without requiring complex architectural changes. On the LVIS v0.5 benchmark, applying SimCal to a ResNet-50 Mask R-CNN improved the overall Mask AP from 18.0 to 23.4. The most dramatic gains were seen in rare classes ($AP_r$), which increased from 2.7 to 16.4.

When integrated with more advanced models like the Hybrid Task Cascade (HTC), the framework achieved 33.4 AP, setting a new state-of-the-art at the time. This simple yet effective calibration strategy was the core of the winning entry for the 2019 LVIS Challenge.

Conclusion

By identifying classification bias as the central challenge in long-tail instance segmentation, this work shifts the focus from complex architectural modifications to simple, decoupled calibration. SimCal demonstrates that even with very few training instances, models can learn to recognize rare objects effectively if the classification head is properly recalibrated.

Summary updated

Abstract

Most existing object instance detection and segmentation models only work well on fairly balanced benchmarks where per-category training sample numbers are comparable, such as COCO. They tend to suffer performance drop on realistic datasets that are usually long-tailed. This work aims to study and address such open challenges. Specifically, we systematically investigate performance drop of the state-of-the-art two-stage instance segmentation model Mask R-CNN on the recent long-tail LVIS dataset, and unveil that a major cause is the inaccurate classification of object proposals. Based on such an observation, we first consider various techniques for improving long-tail classification performance which indeed enhance instance segmentation results. We then propose a simple calibration framework to more effectively alleviate classification head bias with a bi-level class balanced sampling approach. Without bells and whistles, it significantly boosts the performance of instance segmentation for tail classes on the recent LVIS dataset and our sampled COCO-LT dataset. Our analysis provides useful insights for solving long-tail instance detection and segmentation problems, and the straightforward \emph{SimCal} method can serve as a simple but strong baseline. With the method we have won the 2019 LVIS challenge. Codes and models are available at https://github.com/twangnh/SimCal.
BibTeX
@inproceedings{wang2020devil,
  author = {Tao Wang and Yu Li and Bingyi Kang and Junnan Li and Jun Hao Liew and Sheng Tang and Steven C. H. Hoi and Jiashi Feng},
  title = {{The Devil Is in Classification: A Simple Framework for Long-Tail Instance Segmentation}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58568-6\_43},
}
Figure