Abstract
Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. Despite the extensive knowledge that vision-language models (VLMs) possess, it remains largely unexplored whether they are capable of performing this fundamental task. To address this, we introduce VLM2-Bench, a benchmark designed to assess whether VLMs can Visually Link Matching cues, with 9 subtasks and over 3,000 test cases. Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with further analysis of various language-side and vision-side prompting methods, leads to a total of eight key findings. We identify critical challenges in models' ability to link visual cues, highlighting a significant performance gap where even GPT-4o lags 34.80% behind humans. Based on these insights, we advocate for (i) enhancing core visual capabilities to improve adaptability and reduce reliance on prior knowledge, (ii) establishing clearer principles for integrating language-based reasoning in vision-centric tasks to prevent unnecessary biases, and (iii) shifting vision-text training paradigms toward fostering models' ability to independently structure and infer relationships among visual cues.
Introduction
Humans possess an innate ability to navigate the world by linking visual cues. If you see a person in a red jacket in one photo and a person in a red jacket in another, you instinctively look for secondary cues—facial features, height, or gait—to determine if they are the same individual. While current Vision-Language Models (VLMs) are incredibly knowledgeable, it remains unclear if they can perform this fundamental "identity matching" task.
Existing benchmarks often allow models to "cheat" by using external knowledge (e.g., recognizing a celebrity) or focusing on broad, abstract comparisons. There is a lack of rigorous testing for explicit matching visual cues—the raw ability to see that "Object A in Image 1" is physically identical to "Object B in Image 2" based solely on visual evidence.
The VLM2-Bench Framework
To address this gap, the authors introduce VLM2-Bench, a benchmark comprising 9 subtasks and over 3,000 test cases. The framework evaluates models across three primary dimensions:
- General Cue (GC): Matching and tracking changes in edited images (e.g., "Was the salad removed in the second frame?").
- Object-centric Cue (OC): Comparing, counting, and grouping everyday objects (cups, pets, bags) across different scenes.
- Person-centric Cue (PC): Identifying the same individual across non-adjacent video frames or different photos using facial features and clothing.
The dataset utilizes "distractor" images—objects or people that look highly similar but are distinct—to ensure models aren't just making lucky guesses. GC data is repurposed from image-editing datasets like MagicBrush, OC data is manually collected from online marketplaces, and PC data is sourced from IMDB and social media, featuring diverse racial groups and genders.
Key Findings and Analysis
The evaluation of 12 state-of-the-art VLMs revealed several critical insights:
- The Massive Human-VLM Gap: Humans achieve ~94.44% accuracy across these tasks. Even the best commercial models, such as GPT-4o (59.56%) and Claude-3.7-sonnet (59.57%), fall significantly short. Strong open-source models like Qwen2.5-VL-7B reach 55.86%, while many others perform near the chance level of ~33.72%.
- People vs. Objects: Models are generally better at linking person-centric cues than object-centric ones. This is likely because VLMs are heavily trained on person-related data where names act as "anchors" for identity.
- Resolution as a Bottleneck: Performance drops sharply as image resolution decreases. This indicates that models require fine-grained pixels to succeed rather than relying on global layouts, yet current architectures struggle to process this detail efficiently.
- Prompting Effects: While Chain-of-Thought (CoT) reasoning generally helps, "Visual Prompting" (like coordinate grids) actually confused smaller 7B parameter models. Only larger models like GPT-4o could effectively leverage spatial grids to improve accuracy.
Conclusion
VLM2-Bench highlights that while VLMs are proficient at "looking," they struggle with "connecting the dots" across visual sequences. To bridge the 34.80% gap between AI and human performance, future research should focus on shifting training paradigms toward "relationship-centric" learning and improving visual encoders to handle fine-grained details without excessive computational overhead. Establishing clearer principles for integrating language-based reasoning into vision tasks is also essential to avoid introducing unnecessary biases.
Summary updated