An Interactive Image Segmentation Method Based on Multi-Level Semantic Fusion

Ruirui Zou, Qinghui Wang, Falin Wen, Yang Chen, Jiale Liu, Shaoyi Du, Chengzhi Yuan2023 · DOI

TL;DR

This paper introduces a novel interactive image segmentation method that leverages multi-level semantic fusion and a channel attention mechanism to achieve more accurate and refined object segmentation. By effectively integrating user guidance with deep learning, the method significantly improves segmentation quality, particularly at object boundaries, while requiring fewer user interactions than existing approaches.

What Problem Does This Paper Address?

Accurate image segmentation is crucial for many applications, from image editing to medical diagnosis. While automatic segmentation methods have advanced, they often struggle with precision, especially for complex objects or fine details. Interactive image segmentation (IIS) offers a solution by allowing users to provide guidance (like clicks or bounding boxes) to refine segmentation results. However, existing IIS methods often fail to effectively utilize this user guidance to inform deep learning models, leading to suboptimal performance. Specifically, they can struggle with information loss during the multi-scale processing inherent in deep networks, resulting in rough segmentation edges and a need for numerous user interactions to achieve satisfactory results.

Methods

The proposed method builds upon the DeepLabv3+ architecture and employs a coarse-to-fine segmentation strategy with two subnets: CoarseNet and FineNet.

  • CoarseNet: This subnet focuses on capturing multi-scale features and establishing an initial segmentation. A key innovation here is the cross-stage feature aggregation module. This module addresses information loss during upsampling and downsampling by propagating multi-scale features from previous stages to the current one, ensuring that each stage has access to richer, prior information for more discriminative representations.
  • FineNet: This subnet refines the segmentation from CoarseNet, aiming for precise object boundaries. It incorporates a feature channel attention mechanism. This mechanism assigns weights to different feature channels based on their importance, allowing the network to focus on informative features and suppress irrelevant ones. This leads to finer segmentation edges. The authors also introduce a method for redistributing feature weights after training to further enhance features that contribute to accuracy.
  • User Guidance: The method utilizes both "internal" (foreground) and "external" (background) guidance points. It simulates these by extracting extreme points of the object from ground truth masks for internal guidance and automatically deducing a bounding box for external guidance. This approach is designed to be robust to slight inaccuracies in user input.

Datasets

The primary dataset used for training and evaluation is the PASCAL Visual Object Classes (VOC) 2012 dataset. The paper mentions that the training and validation datasets contain a specific number of images and annotated objects across 20 object classes. No new benchmarks were introduced.

Key Insights and Ideas

  • Bridging the Gap in IIS: The core insight is that effectively fusing multi-level semantic information and selectively attending to feature channels are critical for improving interactive segmentation, especially in addressing the limitations of existing methods.
  • Mitigating Information Loss: The cross-stage feature aggregation module is a clever way to combat the inherent information degradation that occurs in deep, multi-scale networks, ensuring that valuable semantic context isn't lost during processing.
  • Refining Boundaries: The channel attention mechanism is shown to be crucial for achieving sharp and precise object boundaries, a common challenge in segmentation tasks.
  • Efficiency Through Interaction: The method demonstrates that achieving high segmentation accuracy doesn't necessarily require a massive number of user interactions, making the process more user-friendly.

Key Results

  • Improved Accuracy: The proposed method achieved an Intersection over Union (IOU) accuracy approximately 2.1% higher than the currently popular interactive image segmentation method on the PASCAL VOC 2012 dataset.
  • Robustness: Experiments showed that the method is robust to random perturbations in user clicks, with the impact on the final result being minimal (0.3% to 0.5%) even with disturbances up to 25 pixels.
  • Efficiency: The method requires only 4 clicks to achieve a high IOU (85% on PASCAL and 96.5% on GrabCut), making it one of the most efficient methods in terms of user interaction.
  • Ablation Studies: Both the cross-stage feature aggregation and channel attention modules individually improved performance, but their joint application yielded the best results, indicating complementary effects.

Limitations and Future Work

  • High Occlusion: The authors acknowledge that the method exhibits insufficient segmentation accuracy for objects with high occlusion ratios. This hinders user interaction and makes complete segmentation challenging. They suggest including training sets with occluded objects to address this.
  • Segmentation Speed: While the method achieves good accuracy, further optimization of its running speed is necessary, especially for interactive applications where waiting time can be a bottleneck.
  • Further Research Directions: The paper suggests including more training data for occluded objects and optimizing the model's speed for real-time applications.

Summary updated

BibTeX
@misc{zou2023interactive,
  author = {Ruirui Zou and Qinghui Wang and Falin Wen and Yang Chen and Jiale Liu and Shaoyi Du and Chengzhi Yuan},
  title = {{An Interactive Image Segmentation Method Based on Multi-Level Semantic Fusion}},
  year = {2023},
  doi = {10.3390/s23146394},
}
Figure