HARIS: Human-Like Attention for Reference Image Segmentation

Mengxi Zhang, Heqing Lian, Yiming Liu, Jie Chen

TL;DR

This paper introduces HARIS, a novel method for referring image segmentation that uses a "Human-Like Attention" mechanism to better focus on the intended object by incorporating a feedback signal. By also employing parameter-efficient fine-tuning (PEFT), HARIS achieves state-of-the-art performance and impressive zero-shot capabilities.

What Problem Does This Paper Address?

Referring image segmentation (RIS) is a challenging task that involves pinpointing a specific object in an image based on a natural language description. Unlike traditional image segmentation, which deals with predefined categories, RIS needs to understand nuanced language and locate arbitrary objects. Existing methods often fuse visual and linguistic features using a bottom-up attention mechanism. While effective, this approach can lead to the model attending to irrelevant image-text pairs, resulting in inaccurate segmentation masks. For instance, when segmenting "the guy in the black jacket," a bottom-up method might incorrectly associate "black" with other dark regions in the image, not just the intended jacket. This paper aims to overcome this by developing a more human-like attention process that refines its focus.

Methods

The core innovation of HARIS lies in its Human-Like Attention (HLA) block. This block introduces a feedback mechanism inspired by human cognition, where new information is used to refine understanding. Specifically, HLA takes multi-modal features and generates a feedback signal that is then fed back into the attention process. This allows the network to re-evaluate and discard irrelevant image-text associations, leading to a more precise focus on the target object.

The HLA block consists of two main branches:

  • Language-Weighted Attention: This branch uses the overall sentence representation to modulate visual tokens, creating language-aware visual tokens.
  • Vision-Weighted Attention: This branch uses visual tokens to modulate linguistic tokens, creating vision-aware linguistic tokens.

Crucially, the feedback signal is derived from these modulated multi-modal tokens and is used as a "second-round input" for the Language-Weighted Attention, enabling iterative refinement.

Additionally, HARIS employs a hierarchical design to leverage visual features from different layers of the image encoder, capturing both semantic and fine-grained details. The Mask Decoder is built upon a standard Transformer Decoder, using a learnable query token alongside vision-aware linguistic tokens to generate the final segmentation mask.

To preserve the strong zero-shot capabilities of pre-trained visual and textual encoders, HARIS utilizes the Parameter-Efficient Fine-Tuning (PEFT) framework. During training, only the parameters of the fusion blocks and the mask decoder are updated, preventing catastrophic forgetting and maintaining the encoders' generalizability.

Datasets

HARIS is evaluated on three widely used RIS benchmarks:

  • RefCOCO
  • RefCOCO+
  • G-Ref

These datasets are sourced from MSCOCO and differ in average expression length and the inclusion of absolute positional terms. The paper also reports zero-shot performance on the PhraseCut dataset, which features a significantly larger number of categories (1287) compared to COCO (80).

Key Insights and Ideas

  • Human-like iterative refinement: The core idea of using a feedback signal to iteratively refine attention is a powerful analogy to human visual cognition, allowing the model to "think twice" and discard distractions.
  • Addressing irrelevant image-text pairs: The bottom-up nature of traditional attention is a significant bottleneck. HARIS's feedback mechanism directly tackles this by actively filtering out spurious correlations.
  • PEFT for zero-shot preservation: Leveraging PEFT is a smart strategy to benefit from powerful pre-trained models without sacrificing their ability to generalize to unseen categories or tasks. This is particularly important for RIS, where diverse expressions and objects are common.
  • Hierarchical features matter: Using features from multiple visual layers, not just the final one, provides a richer representation that aids in both understanding the overall scene and pinpointing precise details.

Key Results

HARIS achieves state-of-the-art (SOTA) performance across all three major RIS datasets (RefCOCO, RefCOCO+, G-Ref), even with frozen visual and text encoders.

  • On RefCOCO, HARIS shows IoU gains of 1.25%, 1.47%, and 0.53% on the val, testA, and testB splits, respectively, surpassing the previous best method (MCRES).
  • On RefCOCO+, it achieves new SOTA performance with improvements of 1.69%, 2.24%, and 0.10% on the respective splits.
  • On the challenging G-Ref dataset, HARIS improves IoU by 1.54%, 0.38%, and 1.75% on the val(U), test(U), and val(G) splits.
  • Furthermore, HARIS demonstrates excellent zero-shot performance on the PhraseCut dataset, significantly outperforming previous methods. For instance, trained on RefCOCO, it achieves 21.62% IoU on PhraseCut, compared to 16.68% for LAVT.

Limitations and Future Work

  • Ablation Study Insights: The ablation studies reveal that the Language-Weighted Attention component is more critical than Vision-Weighted Attention for performance gains. Removing the feedback branch (w/o FB) resulted in a noticeable IoU drop, highlighting its importance.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success of the "Human-Like Attention" concept suggests potential applications in other vision-language tasks where nuanced understanding and focus are crucial. The PEFT approach also opens doors for more efficient adaptation of large models to specialized tasks.

Summary updated

Abstract

Referring image segmentation (RIS) aims to locate the particular region corresponding to the language expression. Existing methods incorporate features from different modalities in a bottom-up manner. This design may get some unnecessary image-text pairs, which leads to an inaccurate segmentation mask. In this paper, we propose a referring image segmentation method called HARIS, which introduces the Human-Like Attention mechanism and uses the parameter-efficient fine-tuning (PEFT) framework. To be specific, the Human-Like Attention gets a feedback signal from multi-modal features, which makes the network center on the specific objects and discard the irrelevant image-text pairs. Besides, we introduce the PEFT framework to preserve the zero-shot ability of pre-trained encoders. Extensive experiments on three widely used RIS benchmarks and the PhraseCut dataset demonstrate that our method achieves state-of-the-art performance and great zero-shot ability.
BibTeX
@misc{zhang2024haris,
  author = {Mengxi Zhang and Heqing Lian and Yiming Liu and Jie Chen},
  title = {{HARIS: Human-Like Attention for Reference Image Segmentation}},
  year = {2024},
  eprint = {2405.10707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.10707},
}
Figure