Mask4Align: Aligned Entity Prompting with Color Masks for Multi-Entity Localization Problems

Haoquan Zhang, Ronggang Huang, Yi Xie, Huaidong ZhangCVPR 2024 · OpenReview

Abstract

Recognizing and localizing specific entities in complex visual scenes is a core challenge for Vision-Language Models (VLMs). While current models excel at general object identification, they often struggle to disambiguate multiple instances of the same category—a challenge termed the multi-entity localization problem. We present Mask4Align, a training-free method that uses color masks as visual prompts to enable off-the-shelf VLMs to precisely identify and localize target entities. By converting spatial localization into a color-discrimination task, Mask4Align achieves state-of-the-art results on gaze target prediction and multi-entity localization benchmarks.

Introduction

Current Visual Question Answering (VQA) systems struggle with "which one" questions. In a scene with multiple similar entities (e.g., several players in identical soccer jerseys), natural language descriptions are often insufficient for precise disambiguation. Humans typically solve this by pointing. Mask4Align introduces a "visual pointing" mechanism for VLMs, allowing them to distinguish between identical-looking entities without requiring additional training or fine-tuning.

Methodology

Mask4Align is a two-stage, training-free framework that leverages the discriminative power of pre-trained VLMs like CLIP or ALBEF.

  1. Target Entity Acquisition: The system first identifies the object category of interest. It uses the Recognize Anything Model (RAM) to generate candidate tags and a VQA module to select the tag (e.g., "player") that matches the user's query.
  2. Multi-Entity Localization: Once the category is identified, the Segment Anything Model (SAM) extracts all instances.
  3. Color Masking: Distinct color masks (e.g., Red, Green, Blue) are overlaid on each instance.
  4. Grayscale Background: The background is desaturated to prevent original image colors from interfering with the model's color perception.
  5. Aligned Prompting: The VQA model is prompted with the original question and color-coded candidate answers (e.g., "the PURPLE player").
Figure 4: The Mask4Align method applies distinct color masks to similar entities (apples) against a grayscale background to facilitate disambiguation.
  1. Recursive Grouping (RG-VQA): For scenes with more entities than available colors, a hierarchical approach uses K-means clustering to group entities. The model iteratively selects the correct group until the specific target is isolated.

Experimental Evaluation

The researchers evaluated Mask4Align on two primary benchmarks:

  • GazeFollow: A standard dataset for predicting gaze targets.
  • ME-VQA (Multi-Entity VQA): A new dataset of 75 images across 5 categories with human-annotated masks, specifically designed to test multi-entity localization.
Figure 6: End-to-end examples of Mask4Align solving localization problems in soccer and forest scenes by mapping questions to color-coded answers.

Results and Discussion

Mask4Align demonstrated significant performance improvements across various base models. On the ME-VQA dataset using ALBEF, the method improved the S-measure ($S_\alpha$) from 0.514 to 0.821 and the F-measure ($F_{max}$) from 0.572 to 0.789. It also outperformed baseline models and alternative strategies like bounding boxes ("Box4Align") on the GazeFollow task.

The core innovation of Mask4Align is the realization that VLMs are highly proficient at identifying basic colors even when they struggle with complex spatial reasoning. By "hacking" this existing knowledge through visual prompting, the method preserves semantic context and achieves high precision in a zero-shot manner.

Conclusion

While effective, the system's performance is currently bounded by the underlying VQA model's capabilities and a fixed color palette. Future work includes dynamic color selection and scaling the ME-VQA dataset to encompass more diverse scenarios.

Summary updated

Abstract

In Visual Question Answering (VQA), recognizing and localizing entities pose significant challenges. Pretrained vision-and-language models have addressed this problem by providing a text description as the answer. However, in visual scenes with multiple entities, textual descriptions struggle to distinguish the entities from the same category effectively. Consequently, the VQA dataset is limited by the limitations of text description and cannot adequately cover scenarios involving multiple entities. To address this challenge, we introduce a Mask for Align (Mask4Align) method, which can determine the entity's position in the given image that best matches the user-input question. This method incorporates colored masks into the image, enabling the VQA model to handle discrimination and localization challenges associated with multiple entities. To process an arbitrary number of similar entities, Mask4Align is designed hierarchically to discern subtle differences, achieving precise localization. Since Mask4Align directly utilizes pre-trained models, it does not introduce additional training overhead. Extensive experiments conducted on both the gaze target prediction task dataset and our proposed multi-entity localization dataset showcase the superiority of Mask4Align. Code and dataset are available at https://github.com/haoquanzhang/mask4align.
BibTeX
@inproceedings{zhang2024mask,
  author = {Haoquan Zhang and Ronggang Huang and Yi Xie and Huaidong Zhang},
  title = {{Mask4Align: Aligned Entity Prompting with Color Masks for Multi-Entity Localization Problems}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=swH8VoK3Vc},
}
Figure