SipMaskv2: Enhanced Fast Image and Video Instance Segmentation

Jiale Cao, Yanwei Pang, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Ling Shao

Abstract

We propose a fast single-stage method for both image and video instance segmentation, called SipMask, that preserves the instance spatial information by performing multiple sub-region mask predictions. The main module in our method is a light-weight spatial preservation (SP) module that generates a separate set of spatial coefficients for the sub-regions within a bounding-box, enabling a better delineation of spatially adjacent instances. To better correlate mask prediction with object detection, we further propose a mask alignment weighting loss and a feature alignment scheme. In addition, we identify two issues that impede the performance of single-stage instance segmentation and introduce two modules, including a sample selection scheme and an instance refinement module, to address these two issues. Experiments are performed on both image instance segmentation dataset MS COCO and video instance segmentation dataset YouTube-VIS. On MS COCO <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">test-dev</monospace> set, our method achieves a state-of-the-art performance. In terms of real-time capabilities, it outperforms YOLACT by a gain of 3.0% (mask AP) under the similar settings, while operating at a comparable speed. On YouTube-VIS validation set, our method also achieves promising results. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JialeCao001/SipMask</uri> .
BibTeX
@article{cao2023sipmaskv,
  author = {Jiale Cao and Yanwei Pang and Rao Muhammad Anwer and Hisham Cholakkal and Fahad Shahbaz Khan and Ling Shao},
  title = {{SipMaskv2: Enhanced Fast Image and Video Instance Segmentation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=3BKfCSxXKZ},
}

Loading related content...

Figure