Abstract
Recognizing and localizing specific entities in complex visual scenes is a core challenge for Vision-Language Models (VLMs). While current models excel at general object identification, they often struggle to disambiguate multiple instances of the same category—a challenge termed the multi-entity localization problem. We present Mask4Align, a training-free method that uses color masks as visual prompts to enable off-the-shelf VLMs to precisely identify and localize target entities. By converting spatial localization into a color-discrimination task, Mask4Align achieves state-of-the-art results on gaze target prediction and multi-entity localization benchmarks.
Introduction
Current Visual Question Answering (VQA) systems struggle with "which one" questions. In a scene with multiple similar entities (e.g., several players in identical soccer jerseys), natural language descriptions are often insufficient for precise disambiguation. Humans typically solve this by pointing. Mask4Align introduces a "visual pointing" mechanism for VLMs, allowing them to distinguish between identical-looking entities without requiring additional training or fine-tuning.
Methodology
Mask4Align is a two-stage, training-free framework that leverages the discriminative power of pre-trained VLMs like CLIP or ALBEF.
- Target Entity Acquisition: The system first identifies the object category of interest. It uses the Recognize Anything Model (RAM) to generate candidate tags and a VQA module to select the tag (e.g., "player") that matches the user's query.
- Multi-Entity Localization: Once the category is identified, the Segment Anything Model (SAM) extracts all instances.
- Color Masking: Distinct color masks (e.g., Red, Green, Blue) are overlaid on each instance.
- Grayscale Background: The background is desaturated to prevent original image colors from interfering with the model's color perception.
- Aligned Prompting: The VQA model is prompted with the original question and color-coded candidate answers (e.g., "the PURPLE player").
- Recursive Grouping (RG-VQA): For scenes with more entities than available colors, a hierarchical approach uses K-means clustering to group entities. The model iteratively selects the correct group until the specific target is isolated.
Experimental Evaluation
The researchers evaluated Mask4Align on two primary benchmarks:
- GazeFollow: A standard dataset for predicting gaze targets.
- ME-VQA (Multi-Entity VQA): A new dataset of 75 images across 5 categories with human-annotated masks, specifically designed to test multi-entity localization.
Results and Discussion
Mask4Align demonstrated significant performance improvements across various base models. On the ME-VQA dataset using ALBEF, the method improved the S-measure ($S_\alpha$) from 0.514 to 0.821 and the F-measure ($F_{max}$) from 0.572 to 0.789. It also outperformed baseline models and alternative strategies like bounding boxes ("Box4Align") on the GazeFollow task.
The core innovation of Mask4Align is the realization that VLMs are highly proficient at identifying basic colors even when they struggle with complex spatial reasoning. By "hacking" this existing knowledge through visual prompting, the method preserves semantic context and achieves high precision in a zero-shot manner.
Conclusion
While effective, the system's performance is currently bounded by the underlying VQA model's capabilities and a fixed color palette. Future work includes dynamic color selection and scaling the ME-VQA dataset to encompass more diverse scenarios.
Summary updated