Semi-Supervised Instance-Segmentation Model for Feature Transfer Based on Category Attention

Hao Wang, Juncai Liu, Changhai Huang, Xuewen Yang, Dasha Hu, Liangyin Chen, Xiaoqing Xing, Yuming Jiang

TL;DR

This paper introduces AFT-Mask, a novel semi-supervised instance segmentation model that leverages category attention to improve feature transfer from source tasks. By using object classification predictions as "attention" signals, AFT-Mask enhances the feature migration process, leading to more accurate instance segmentation with reduced training time compared to existing methods.

What Problem Does This Paper Address?

Instance segmentation, a crucial task in areas like autonomous driving and medical imaging, typically requires large amounts of labeled data, making fully supervised approaches costly and impractical. While semi-supervised methods offer a solution by utilizing less labeled data, existing approaches often struggle with either insufficient feature utilization from source tasks (in transfer learning) or low segmentation accuracy (in pseudo-label generation). This paper specifically targets the limitations of transfer learning-based semi-supervised instance segmentation, aiming to improve performance by more effectively leveraging source task characteristics.

Methods

The core of the proposed AFT-Mask model is its category attention mechanism integrated into a feature migration optimization module. This module takes the object classification predictions from the detection branch as "attention" signals. Unlike traditional one-hot encodings, this approach uses floating-point classification scores, which are argued to better retain feature information and avoid sparsity.

The model is built upon the Mask R-CNN architecture. A key innovation is the migration-optimization module that connects the feature migration and classification prediction branches. This module uses a sigmoid activation function for standardization, ensuring consistency with the segmentation prediction branch, unlike some prior methods that used softmax. This design aims to enhance segmentation prediction accuracy by effectively transferring knowledge.

To address training time concerns, AFT-Mask employs an inverse-convolutional module to link the feature migration and segmentation prediction modules. This allows the object detection branch's backpropagation to directly benefit the segmentation prediction, potentially speeding up the overall training process. The paper also details specific implementation choices, such as using 1x1 convolution kernels for channel aggregation and bilinear upsampling to reduce parameters in the attention function.

Datasets

The experiments were conducted on the COCO2017 dataset. For semi-supervised training, the authors created a specific dataset division:

  • Sub-training set A: 25% of the COCO training set with complete instance segmentation labels.
  • Sub-training set B: An additional 25% of the COCO training set with only category and border annotations.

This combined dataset (referred to as the "COCO-25% semi-supervised dataset") was used for training, and the COCO instance segmentation verification set served as the test dataset.

Key Insights and Ideas

  • Category Classification as Attention: Using the output of object classification directly as "attention" is a clever way to guide feature transfer. It's more informative than sparse one-hot vectors and directly relates to the target categories.
  • Bridging Feature Transfer and Prediction: The migration-optimization module is crucial for effectively connecting the knowledge learned from source tasks (feature transfer) with the actual segmentation task, leading to improved accuracy.
  • Standardization for Compatibility: Aligning the activation function (sigmoid) in the attention module with the segmentation prediction branch is a practical necessity that improves model integration and performance.
  • Channel Multiplication for Richer Information: The paper argues that using element-wise multiplication (instead of addition) to combine feature maps from different branches provides richer information, contributing to better attention feature maps.
  • Efficient Training through Shared Gradients: The inverse-convolutional module's role in allowing gradient flow from segmentation to detection is a smart design choice for potentially reducing training time.

Key Results

The AFT-Mask model demonstrated improved performance in semi-supervised instance segmentation.

  • Network Structure Influence: Experiments showed that the AFT-Mask model with a complete structure (attention + inverse convolution) and using channel multiplication (AFT-Mask att+deconv(multiply)) achieved the best results, outperforming variants with only attention or additive connections. Specifically, it achieved an mAP of 11.1% and AP50 of 28.0%.
  • Comparison with Benchmarks: When compared against Mask R-CNN and a class-agnostic Mask R-CNN on the COCO-25% semi-supervised dataset, AFT-Mask showed higher performance across several metrics, including mAP, AP50, and AP75. It also performed competitively with the Mask×R-CNN model.

Limitations and Future Work

  • Category Generalization: The authors acknowledge that while AFT-Mask performs well, there's still a gap in performance compared to Mask×R-CNN, particularly concerning category generalization. This suggests room for improvement in how the model handles diverse object categories.
  • Dataset Size: The study used a relatively small portion (25%) of the COCO training set for instance segmentation labels, which might limit the overall achievable accuracy.
  • Future Directions: The authors suggest future work could explore:
  • Investigating the impact of different annotation types (category, border, supervised) on model performance.
  • Designing more general and efficient semi-supervised instance segmentation models.
  • Developing category-independent feature migration and related optimization algorithms to enhance category generalization.

Summary updated

Abstract

In the task of image instance segmentation, semi-supervised instance segmentation algorithms have received constant research attention over recent years. Among these algorithms, algorithms based on transfer learning are better than algorithms based on pseudo-label generation in terms of segmentation performance, but they can not make full use of the relevant characteristics of source tasks. To improve the accuracy of these algorithms, this work proposes a semi-supervised instance segmentation model AFT-Mask (attention-based feature transfer Mask R-CNN) based on category attention. The AFT-Mask model takes the result of object-classification prediction as “attention” to improve the performance of the feature-transfer module. In detail, we designed a migration-optimization module for connecting feature migration and classification prediction to enhance segmentation-prediction accuracy. To verify the validity of the AFT-Mask model, experiments were conducted on two types of datasets. Experimental results show that the AFT-Mask model can achieve effective knowledge transfer and improve the performance of the benchmark model on semi-supervised instance segmentation.
BibTeX
@inproceedings{wang2022semi,
  author = {Hao Wang and Juncai Liu and Changhai Huang and Xuewen Yang and Dasha Hu and Liangyin Chen and Xiaoqing Xing and Yuming Jiang},
  title = {{Semi-Supervised Instance-Segmentation Model for Feature Transfer Based on Category Attention}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=vkP981TeR9},
}

Loading related content...

Figure