Object Aware Contrastive Prior for Interactive Image Segmentation

Praful Mathur, Shashi Kumar Parwani, Mrinmoy Sen, Roopa Sheshadri, Aman SharmaWACV 2024

TL;DR

This paper introduces Object Aware Contrastive Embeddings (OACE), a novel way to represent user clicks in interactive image segmentation. By training a prior network using contrastive learning, OACE captures robust foreground object features, significantly improving single-click segmentation accuracy and reducing user effort. The proposed MSFNet architecture effectively leverages OACE, outperforming existing state-of-the-art methods by 21% on challenging datasets.

What Problem Does This Paper Address?

Interactive image segmentation aims to allow users to precisely segment objects in images with minimal input. Traditional methods often rely on iterative positive/negative clicks to refine segmentation masks, which can be tedious and lead to a poor user experience. A key challenge is achieving high accuracy with a single click, especially for complex objects or when user clicks are near object boundaries. Existing methods struggle with this due to how user clicks are represented; typically, simple distance transforms or fixed-radius disks are used. These representations are highly sensitive to the exact click location, leading to inconsistent segmentation outputs and a lack of robustness. Furthermore, many models are trained on datasets biased towards frequent object categories, leading to poor performance on rare or unseen objects.

Methods

The paper proposes two main components:

  • Object Aware Contrastive Embeddings (OACE): This is a novel representation for user clicks. It's generated by a contrastive prior network, which takes an image and a user click (represented as a binary disk) as input. The network is trained using a contrastive loss function that maximizes similarity between intra-object features and minimizes similarity between object and background features. This process learns to extract rich, object-aware foreground features that are robust to variations in click location and distinct from background features.
  • Multi-Stage Fusion Interactive Segmentation Network (MSFNet): This is the core segmentation network that utilizes OACE. It employs a standard HRNet18 backbone with a CNN-based segmentation head. The key innovation here is the "Multi-Stage Fusion" (MSF) module. Instead of fusing the image and OACE inputs early on, the MSF module injects OACE information at multiple stages within the HRNet backbone. This helps prevent the dilution of user click information during processing. The MSF module itself consists of convolution blocks and Squeeze-and-Excitation (SE) inception blocks for adaptive channel recalibration.

Datasets

The paper uses the following datasets:

  • Training: COCO+LVIS (combined for training the OACE prior network and MSFNet).
  • Evaluation:
    • GrabCut: 50 images, single object mask per image.
  • Berkeley: 96 images, 100 ground truth masks.
  • SBD (Semantic Boundaries Dataset): Validation set with 6671 instance-level masks from 2820 images.
  • DAVIS: 345 randomly sampled frames from video sequences.
  • Custom Test Sets: Two custom test sets were created with 100 images each, specifically for evaluating performance on rare and frequent object categories, based on LVIS dataset categorization.

Key Insights and Ideas

  • OACE for Robust Click Representation: The core insight is that representing user clicks not just as a location but as an "object-aware feature embedding" derived from contrastive learning significantly improves robustness. This embedding captures the essence of the object around the click, making the segmentation less sensitive to the precise pixel clicked.
  • Contrastive Learning for Feature Distinction: Applying contrastive learning to learn features that are similar within an object but dissimilar to the background is crucial. This helps the model inherently understand what belongs to the object and what doesn't, even for unseen categories.
  • Multi-Stage Fusion for Information Preservation: Injecting the OACE information at multiple stages within the network (MSF module) is more effective than early fusion. This ensures that the valuable user click context is consistently leveraged throughout the feature extraction process.
  • Addressing Rare Categories: The proposed OACE approach shows promise in improving segmentation for rare object categories, which are often a weak point for models trained on imbalanced datasets. OACE helps the network develop a better understanding of object features even when they are underrepresented in training.

Key Results

  • Improved Single-Click Accuracy: The proposed MSFNet+OACE method achieves a significant improvement in single-click segmentation accuracy (mIoU@1). It outperforms existing state-of-the-art methods by 21% on average.
  • Reduced Clicks Needed: The standard metric of Number of Clicks (NoC@90) shows that the proposed method requires fewer clicks to achieve 90% IoU, indicating greater efficiency.
  • Comparable to Large Models: MSFNet+OACE, with only 15.2M parameters, achieves performance comparable to the much larger Segment Anything Model (SAM) (632M parameters) on mIoU@1, despite being trained on a smaller dataset.
  • Effectiveness on Rare Categories: The custom test sets reveal that MSFNet+OACE improves segmentation accuracy on rare object categories by approximately 20%, compared to only a 7% improvement on frequent categories.
  • Efficiency: The model is lightweight (15.2M parameters, ~15MB TFLite size) and efficient, with inference times of around 250ms on a modern smartphone, making it suitable for embedded devices.

Limitations and Future Work

  • Thin Object Segmentation: The authors acknowledge that segmenting very thin objects accurately remains a challenge. This is attributed to the network processing images at a relatively low resolution (512x512), which can cause fine details like hair strands to be missed.
  • Future Work: The authors suggest exploring high-resolution post-processing or matting-based solutions to improve the segmentation of thin objects.

Summary updated

BibTeX
@inproceedings{mathur2024object,
  author = {Praful Mathur and Shashi Kumar Parwani and Mrinmoy Sen and Roopa Sheshadri and Aman Sharma},
  title = {{Object Aware Contrastive Prior for Interactive Image Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure