Weakly Supervised Instance Segmentation by Deep Multi-Task Community Learning

Seohyun Kim, Jaedong Hwang, Jeany Son, Bohyung Han

Abstract

We present a weakly supervised instance segmentation algorithm based on deep community learning with multiple tasks. This task is formulated as a combination of weakly supervised object detection and semantic segmentation, where individual objects of the same class are identified and segmented separately. We address this problem by designing a unified deep neural network architecture, which has a positive feedback loop of object detection with bounding box regression, instance mask generation, instance segmentation, and feature extraction. Each component of the network makes active interactions with others to improve accuracy, and the end-to-end trainability of our model makes our results more robust and reproducible. The proposed algorithm achieves state-of-the-art performance in the weakly supervised setting without any additional training such as Fast R-CNN and Mask R-CNN on the standard benchmark dataset. The implementation of our algorithm is available on the project webpage: https://cv.snu.ac.kr/research/WSIS_CL.

Introduction

Instance segmentation, the task of identifying individual objects and their precise pixel-level boundaries, typically requires extensive datasets with manually annotated masks. Weakly Supervised Instance Segmentation (WSIS) aims to overcome this by using only image-level labels. However, WSIS models often suffer from "hot spot" detections, where the network focuses only on the most discriminative parts of an object (e.g., a cat's ears) rather than the entire body. Furthermore, combining detection and segmentation often leads to noise propagation between tasks.

Deep Community Learning Framework

The proposed "Deep Community Learning" framework addresses these issues by treating object detection, mask generation, and segmentation as a collaborative community.

The Deep Community Learning framework architecture, illustrating the positive feedback loop between detection, mask generation, and segmentation modules.

The architecture consists of four primary components:

  1. Feature Extractor: A shared ResNet50 backbone provides features to all subsequent modules.
  2. Object Detection Module: This module identifies object locations and generates pseudo-labels. It utilizes a class-agnostic bounding box regressor, which helps the model learn general object boundaries and reduces bias toward specific discriminative features.
  3. Instance Mask Generation (IMG) Module: Using Class Activation Maps (CAMs), this module creates pseudo-ground-truth masks. It incorporates "feature smoothing" to penalize extreme activation peaks, forcing the model to attend to the entire object region.
  4. Instance Segmentation Module: This module predicts the final pixel-wise masks based on the refined outputs of the community.

The end-to-end trainability of this system allows for a positive feedback loop where each task informs and improves the others, leading to more robust feature extraction.

Experimental Results

The model was evaluated on the PASCAL VOC 2012 benchmark, using the augmented training set of 10,582 images with only image-level labels. The algorithm achieved a state-of-the-art [email protected] of 38.1 on the validation set.

Qualitative results on the PASCAL VOC 2012 dataset, demonstrating the model's ability to segment entire object bodies across various categories.

Qualitative analysis demonstrates that the community learning approach effectively captures full object bodies, such as the entire wingspan of a bird or the torso of a sheep, outperforming baseline weakly supervised detectors that often produce fragmented masks. Notably, the model achieves these results without requiring complex multi-stage retraining processes typical of other WSIS methods.

Conclusion

Deep Multi-Task Community Learning provides a powerful framework for WSIS by leveraging the interactions between detection and segmentation. While the model shows significant improvements, challenges remain in handling heavily overlapping objects and capturing fine details like small limbs. Future research will focus on improving boundary refinement and handling dense scenes.

Summary updated

Abstract

We present a weakly supervised instance segmentation algorithm based on deep community learning with multiple tasks. This task is formulated as a combination of weakly supervised object detection and semantic segmentation, where individual objects of the same class are identified and segmented separately. We address this problem by designing a unified deep neural network architecture, which has a positive feedback loop of object detection with bounding box regression, instance mask generation, instance segmentation, and feature extraction. Each component of the network makes active interactions with others to improve accuracy, and the end-to-end trainability of our model makes our results more robust and reproducible. The proposed algorithm achieves state-of-the-art performance in the weakly supervised setting without any additional training such as Fast R-CNN and Mask R-CNN on the standard benchmark dataset. The implementation of our algorithm is available on the project webpage: https://cv.snu.ac.kr/research/WSIS_CL.
BibTeX
@inproceedings{kim2021weakly,
  author = {Seohyun Kim and Jaedong Hwang and Jeany Son and Bohyung Han},
  title = {{Weakly Supervised Instance Segmentation by Deep Multi-Task Community Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=HSODaljr9TN},
}
Figure