Weakly Supervised Instance Segmentation Using Hybrid Networks

Shisha Liao, Yongqing Sun, Chenqiang Gao, Pranav Shenoy K. P, Song Mu, Jun Shimamura, Atsushi Sagata

TL;DR

This paper tackles the challenge of instance segmentation with limited annotations by proposing a novel hybrid network. Their approach effectively handles both large and small objects by employing separate branches, leading to significant performance improvements over existing weakly supervised methods.

What Problem Does This Paper Address?

Instance segmentation, a crucial task in computer vision for identifying and delineating individual object instances, typically requires dense pixel-level annotations. Creating these detailed masks is extremely labor-intensive and costly. To mitigate this, researchers are exploring "weakly supervised" methods that leverage less precise annotations like bounding boxes. However, a common pipeline involves generating initial masks from these weaker labels, which often results in a significant number of "invalid" masks, particularly for small or dim objects. Directly training on these noisy masks can degrade performance. This paper addresses this critical issue by proposing a method to effectively handle these invalid masks and improve overall instance segmentation accuracy.

Methods

The core of this paper's contribution is a hybrid network designed to address the problem of invalid masks generated from weakly supervised signals. The proposed architecture consists of two main branches:

  • Large Object Instance Branch: This branch is trained using the "valid" initial masks (those with a high Intersection over Union with ground truth, though ground truth is not used directly for training). It's built upon a Mask R-CNN framework, enhanced with an Enhanced-FPN module. The Enhanced-FPN aims to improve feature fusion and localization accuracy by balancing the enhancement of different feature maps and shortening the path from low-level to high-level features. This branch handles the majority of well-formed object instances.
  • Small Object Instance Branch: This branch is specifically designed to handle the "invalid" initial masks, which are predominantly small and dim objects. It operates sequentially, first performing object detection (using Faster R-CNN) and then generating segmentation masks. This sequential approach is intended to avoid omissions of small objects.

During training, initial masks are generated using GrabCut and then split into valid and invalid sets based on an IoU threshold (0.5). In testing, results from both branches are fused, with objects smaller than 64x64 pixels attributed to the small object branch and larger objects to the large object branch.

Datasets

  • PASCAL VOC 2012: Used for both training and evaluation. The authors augment the training set with additional images from the SBD dataset, resulting in a total of 10,582 training images. Evaluation is performed on the validation set, which contains 1,449 images.

Key Insights and Ideas

  • Invalid Masks are a Major Hurdle: The paper strongly emphasizes that directly training on automatically generated masks, especially those with a high proportion of invalid ones (due to small object instances), is detrimental to performance.
  • Small Objects are the Achilles' Heel: The analysis clearly shows that invalid masks disproportionately affect small objects, making them a critical area for improvement in weakly supervised segmentation.
  • Hybrid Approach for Robustness: Separating the handling of large, well-defined objects from small, potentially noisy ones into distinct network branches is a key strategy for achieving better overall performance.
  • Enhanced-FPN for Better Feature Representation: The proposed Enhanced-FPN module is crucial for improving the feature extraction process, particularly for localization accuracy, by addressing limitations in conventional FPNs.

Key Results

  • The proposed hybrid network significantly outperforms state-of-the-art weakly supervised instance segmentation methods on the PASCAL VOC 2012 dataset across various metrics, including [email protected], [email protected], and Average Best Overlap (ABO).
  • Specifically, the method achieves an [email protected] of 51.3 and an [email protected] of 22.4, surpassing other box-level methods like DeepMask (39.4, 8.1) and DeepLabBOX (44.8, 16.3).
  • The paper highlights significant improvements in segmenting small objects, which is a key advantage over methods that treat all objects uniformly.
  • Qualitative results (Figure 5) visually demonstrate the superiority of the hybrid method, particularly in capturing small object instances that are missed or poorly segmented by a single-branch approach.

Limitations and Future Work

  • Limitations: The paper acknowledges that the quality of initial masks is still a limiting factor, and the proposed method aims to mitigate this rather than eliminate it entirely. The reliance on GrabCut for initial mask generation, while effective, is still a heuristic.
  • Future Work: Not explicitly detailed in the provided text, but the success of the hybrid approach suggests further exploration into more robust initial mask generation techniques and potentially more sophisticated fusion strategies between the two branches. The authors also mention that their weakly supervised method with bounding box information achieves performance close to supervised methods with precise pixel masks, hinting at future work to bridge this gap further.

Summary updated

Abstract

Weakly-supervised instance segmentation, which could greatly save labor and time cost of pixel mask annotation, has attracted increasing attention in recent years. The commonly used pipeline firstly utilizes conventional image segmentation methods to automatically generate initial masks and then use them to train an off-the-shelf segmentation network in an iterative way. However, the initial generated masks usually contains a notable proportion of invalid masks which are mainly caused by small object instances. Directly using these initial masks to train segmentation model is harmful for the performance. To address this problem, we propose a hybrid network in this paper. In our architecture, there is a principle segmentation network which is used to handle the normal samples with valid generated masks. In addition, a complementary branch is added to handle the small and dim objects without valid masks. Experimental results indicate that our method can achieve significantly performance improvement both on the small object instances and large ones, and outperforms all state-of-the-art methods.
BibTeX
@inproceedings{liao2019weakly,
  author = {Shisha Liao and Yongqing Sun and Chenqiang Gao and Pranav Shenoy K. P and Song Mu and Jun Shimamura and Atsushi Sagata},
  title = {{Weakly Supervised Instance Segmentation Using Hybrid Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682309},
}
Figure