ESAMask: Real-Time Instance Segmentation Fused with Efficient Sparse Attention

Qian Zhang, Lu Chen, Mingwen Shao, Hong Liang, Jie Ren

TL;DR

This paper introduces ESAMask, a novel real-time instance segmentation model that achieves a superior balance between accuracy and speed. By integrating an efficient sparse attention mechanism with a lightweight architecture, ESAMask effectively addresses the computational challenges of traditional attention-based methods while maintaining high segmentation performance.

What Problem Does This Paper Address?

Instance segmentation, a crucial task in computer vision, involves identifying and delineating each distinct object instance within an image at a pixel level. While existing methods have achieved impressive accuracy, they often come with high computational costs and slow inference speeds, making them impractical for real-time applications. This paper aims to bridge this gap by developing a model that is both accurate and efficient, specifically targeting scenarios where real-time performance is critical, such as autonomous driving and industrial automation. The authors highlight the trade-off between accuracy and speed in current segmentation models and seek to optimize this balance.

Methods

ESAMask is built upon a single-stage, anchor-free architecture, prioritizing efficiency. The core innovations lie in three proposed modules:

  • Related Semantic Perceived Attention (RSPA): This module introduces a dynamic and sparse attention mechanism. Instead of applying attention globally, RSPA adaptively identifies and focuses on semantically relevant sub-regions within a feature map. It achieves this by first calculating semantic correlations between regions and then performing fine-grained attention only on the most relevant ones, significantly reducing computational cost.
  • GSInvSAM: This module is designed for the network's neck, aiming to improve feature fusion across different scales while reducing redundancy. It combines GSConv (a lightweight convolutional block) with an inverted bottleneck structure and SimAM (a simple attention module) to enhance channel interaction and capture important feature representations without adding significant parameters.
  • Mixed Receptive Field Context Perception Module (MRFCPM): Integrated into the prototype branch, MRFCPM enables the model to capture feature representations at multiple scales (global, regional, and local). This is crucial for accurately segmenting objects of varying sizes. It achieves this by fusing information from global attention (using GCAM), regional attention (using LKRAM with large kernels), and channel attention (SE block).

Datasets

The primary dataset used for training and evaluation in this paper is MS COCO2017. This dataset is a standard benchmark for instance segmentation tasks, containing a diverse range of images with 80 object categories. The model is trained on the train2017 split (118k images) and evaluated on the val2017 split (5k images).

Key Insights and Ideas

  • Sparse Attention for Efficiency: The paper demonstrates that by making attention mechanisms sparse and semantically aware (RSPA), significant computational savings can be achieved without sacrificing accuracy. This moves beyond fixed window-based sparse attention by dynamically adapting to the content.
  • Multi-Scale Feature Fusion is Key: The MRFCPM module highlights the importance of effectively combining global, regional, and local features to handle objects of diverse scales. This multi-pronged approach to feature extraction is crucial for robust instance segmentation.
  • Lightweight Design Principles: The consistent application of lightweight modules like GSInvSAM and the overall single-stage architecture underscores the paper's commitment to achieving real-time performance. This focus on efficiency is as important as accuracy.
  • Synergistic Module Effects: The ablation studies show that combining the proposed modules (RSPA, GSInvSAM, MRFCPM) leads to a greater improvement in performance than individual modules, indicating their complementary nature.

Key Results

ESAMask achieves a strong performance on the MS COCO dataset, demonstrating a favorable accuracy-speed trade-off:

  • Accuracy: The model achieves a mask AP of 45.4%.
  • Speed: It operates at a frame rate of 45.2 FPS.
  • Comparison: ESAMask outperforms many state-of-the-art methods in terms of the accuracy-speed trade-off. For instance, it achieves higher mAP than YOLOv8-seg (45.4% vs. 42.6%) while maintaining a competitive speed (45.2 FPS vs. 47.8 FPS). It also shows improvements over other sparse attention methods like NA and DiNA.
  • Qualitative Results: Visualizations show ESAMask producing smoother and more detailed segmentation masks, particularly for small and occluded objects, as well as edge regions, compared to other methods.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Increased Parameters and Memory Access: The RSPA module, while efficient, introduces some increase in parameters and memory access frequency due to the computation of the adjacency matrix.
  • Increased Complexity: The MRFCPM module, by modeling global, regional, and local information, adds to the model's complexity and computational cost.

For future work, the authors plan to:

  • Explore Lightweighting Techniques: Investigate pruning and quantization to further enhance the efficiency of sparse attention approaches.
  • Address Challenging Conditions: Extend the segmentation capabilities to handle images captured in adverse weather conditions like rain or fog.

Summary updated

BibTeX
@inproceedings{zhang2023esamask,
  author = {Qian Zhang and Lu Chen and Mingwen Shao and Hong Liang and Jie Ren},
  title = {{ESAMask: Real-Time Instance Segmentation Fused with Efficient Sparse Attention}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=LKNKVMmqBM},
}
Figure