Semi-supervised Active Learning for Instance Segmentation via Scoring Predictions

Changsheng Li 李长升, Jun Wang, Shaoguo Wen, Jianghua Yu, Kaixing Chen, Xin Zhou, Peng Gao, Guotong Xie, Changsheng Li

TL;DR

This paper introduces a novel semi-supervised active learning framework for instance segmentation, a complex task with high annotation costs. By developing a "Triplet Scoring Prediction" (TSP) method that considers class, bounding box, and mask quality, and combining it with a progressive pseudo-labeling strategy, the authors significantly reduce the need for human labeling while achieving state-of-the-art performance.

What Problem Does This Paper Address?

Instance segmentation, which involves identifying and delineating individual objects within an image, is a crucial but labor-intensive task. Unlike simpler tasks like image classification or object detection, instance segmentation requires pixel-level precision for each object, leading to significantly higher annotation costs. While active learning (AL) has been successful in reducing annotation efforts for other computer vision tasks by intelligently selecting informative samples for labeling, its application to instance segmentation has been largely overlooked. Furthermore, many existing AL methods fail to leverage the vast amount of unlabeled data available, a gap that semi-supervised learning (SSL) techniques can address. This paper tackles the challenge of efficiently training high-performance instance segmentation models by combining the strengths of AL and SSL to minimize human annotation while maximizing model accuracy.

Methods

The proposed framework integrates two core components:

  • Triplet Scoring Prediction (TSP): This novel uncertainty sampling strategy explicitly considers the quality of predicted class probabilities, bounding boxes, and masks for each instance. It achieves this by adding two new parallel branches to the standard RCNN and Mask RCNN heads. These branches predict the Intersection over Union (IoU) of the predicted bounding box and mask with their respective ground truths. The TSP score for an instance is then derived from these three scores (class, bounding box IoU, mask IoU) to provide a more comprehensive measure of uncertainty than methods relying solely on classification scores. This allows for more informed selection of informative samples for active learning.
  • Progressive Pseudo-Labeling: To further leverage unlabeled data, the framework employs a progressive pseudo-labeling regime. After an initial model is trained on labeled data, it's used to predict labels for unlabeled data. Instances with high confidence scores (based on the TSP) are then "pseudo-labeled" and added to the training set for the next iteration. A novel loss function is designed to mitigate the noise introduced by these pseudo-labels, particularly by focusing on classifying positive regions rather than entire images to avoid misclassifying background as objects. This iterative process allows the model to learn from both human-annotated and confidently predicted unlabeled data.

Datasets

The proposed method was evaluated on two large medical image datasets:

  • Head CT scans for intracranial hemorrhage (ICH) segmentation: This dataset contains ICH and nine other types of annotations, with 10,000 images for training and 4,154 for testing.
  • Retinal OCT dataset for Edema Lesions segmentation: This dataset, sourced from the AI Challenger 2018 competition, includes 12,800 OCT images with annotations for retinal edema areas.

Key Insights and Ideas

  • Instance-level uncertainty is crucial for instance segmentation AL: Unlike classification, where class probability entropy is sufficient, instance segmentation requires considering bounding box and mask quality for accurate uncertainty estimation.
  • Combining AL and SSL is powerful: The synergy between actively selecting informative samples and leveraging confidently predicted unlabeled data (pseudo-labeling) leads to significant performance gains.
  • Pseudo-labeling needs careful handling: Naive pseudo-labeling can introduce noise and confirmation bias. The proposed method addresses this by using TSP for selection and a modified loss function to focus on positive regions.
  • Instance-level sampling is more effective than image-level sampling for high-cost tasks: For instance segmentation, where annotation is expensive, focusing on individual instances provides better returns than selecting entire images.

Key Results

  • The proposed Triplet Scoring Prediction (TSP) strategy outperformed existing methods like Class Entropy and Learning Loss, achieving approximately 1.0 mAP gain on the CT dataset.
  • The full semi-supervised active learning framework (Semi-supervised TSP) achieved a significant gain of up to 10 mAP compared to random sampling at the second cycle.
  • For the ICH segmentation task, the method reduced the number of labeled images from 3000 to 1500, saving approximately 4500 minutes and $10,500 in annotation costs, while achieving comparable performance to training with the full dataset.
  • Ablation studies confirmed the effectiveness of both the TSP and the pseudo-labeling components, with the mask score branch contributing the most to performance improvement.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the proposed method significantly reduces annotation costs, it still requires some human labeling. The effectiveness of the pseudo-labeling strategy is sensitive to the balancing coefficient $\beta$, which needs careful tuning.
  • Future Work: The authors suggest exploring additional imaging tasks, such as histopathological image segmentation, and applying the method to natural images. They also plan to investigate further improvements in active learning and semi-supervised learning strategies.

Summary updated

Abstract

Active learning generally involves querying the most representative samples for human labeling, which has been widely studied in many fields such as image classification and object detection. However, its potential has not been explored in the more complex instance segmentation task that usually has relatively higher annotation cost. In this paper, we propose a novel and principled semi-supervised active learning framework for instance segmentation. Specifically, we present an uncertainty sampling strategy named Triplet Scoring Predictions (TSP) to explicitly incorporate samples ranking clues from classes, bounding boxes and masks. Moreover, we devise a progressive pseudo labeling regime using the above TSP in semi-supervised manner, it can leverage both the labeled and unlabeled data to minimize labeling effort while maximize performance of instance segmentation. Results on medical images datasets demonstrate that the proposed method results in the embodiment of knowledge from available data in a meaningful way. The extensive quantitatively and qualitatively experiments show that, our method can yield the best-performing model with notable less annotation costs, compared with state-of-the-arts.
BibTeX
@inproceedings{2020semi,
  author = {Changsheng Li 李长升 and Jun Wang and Shaoguo Wen and Jianghua Yu and Kaixing Chen and Xin Zhou and Peng Gao and Guotong Xie and Changsheng Li},
  title = {{Semi-supervised Active Learning for Instance Segmentation via Scoring Predictions}},
  booktitle = {BMVC},
  year = {2020},
}

Loading related content...

Figure