VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues

Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, Yi R. Fung

TL;DR

Researchers have introduced VLM2-Bench, a new benchmark revealing that even state-of-the-art Vision-Language Models (VLMs) struggle with a fundamental human skill: linking matching visual cues across different images or video frames. While humans perform these tasks with near-perfect accuracy, the best models lag behind by over 30%, often failing to recognize the same object or person when viewed from different angles or in new contexts.

What Problem Does This Paper Address?

Humans possess an innate ability to "link" visual information. If you see a person in a black jacket in one photo and the same person in a different setting later, you recognize them by matching facial features or clothing cues, even if you don't know who they are.

Current VLM benchmarks often miss this nuance. As shown in Figure 1, existing evaluations like MMDU or MIRB either don't require explicit linking, rely too heavily on external knowledge (e.g., "Is this the same city?"), or focus on broad, abstract comparisons. The authors argue that for a VLM to truly reason about the world, it must be able to independently structure and infer relationships between specific visual cues across non-adjacent frames or disparate images.

Previous benchmarks vs. VLM2-Bench

Methods

The authors developed VLM2-Bench, which categorizes visual linking into three subsets:

  1. General Cue (GC): Focuses on instance-level changes like adding, removing, or swapping objects between two images.
  2. Object-centric Cue (OC): Tests the ability to track specific everyday objects (e.g., a specific plush toy or shoe) across multiple images.
  3. Person-centric Cue (PC): Evaluates the identification of individuals across different scenes, including a "Video Identity Describing" task.

To ensure high-quality data, they used a semi-automated pipeline. They generated captions using a VLM, filtered for "challenging" cases using a saliency score (calculated via a language model like Llama-3), and then used an LLM to generate QA pairs. Crucially, they implemented consistency-pair validation: to be marked correct on a binary task, a model must correctly answer both a statement (e.g., "X is Y") and its negation ("X is not Y"), effectively neutralizing model-specific biases.

Overview of VLM2-Bench categories and subtasks

Datasets

The primary contribution is the VLM2-Bench dataset, consisting of 3,060 question-answer pairs across 9 subtasks. The data was sourced from existing image editing datasets (for GC) and manual collections from retail sites and social media (for OC and PC) to ensure diverse, real-world visual cues.

The researchers evaluated 12 prominent models, including eight open-source (e.g., Qwen2.5-VL, InternVL2.5) and three commercial heavyweights (GPT-4o, Claude-3.7-Sonnet, and Gemini-2.0-Flash).

Key Insights and Ideas

  • The Human-Model Gap: Humans achieved ~94% accuracy across the benchmark, while the best model (GPT-4o) hovered around 60%. Many open-source models performed near chance level, highlighting a massive deficiency in current visual reasoning.
  • People are Easier than Objects: Models were consistently better at linking person-centric cues than object-centric ones. The authors hypothesize that training data often includes names as "anchors" for people, whereas objects are usually referred to by generic category names, making fine-grained differentiation harder for the model.
  • Resolution Matters: A "Visual Bias Sanity Check" showed that model performance drops sharply as image resolution decreases. This confirms that the benchmark successfully probes fine-grained visual perception rather than allowing models to rely on "shortcuts" like global layout or coarse semantics.
  • The Double-Edged Sword of Prompting: While Chain-of-Thought (CoT) reasoning generally helped, visual prompting (like overlaying a coordinate grid, as seen in Figure 5) actually hurt smaller models. These models struggled to interpret the complex spatial instructions, leading to "faulty reasoning" despite identifying the correct visual elements.
Visual prompting approaches used in the study

Key Results

The evaluation yielded eight key findings, but the quantitative results in Table 1 tell the clearest story:

  • Top Performers: GPT-4o (2024-08-06) and Claude-3.7-Sonnet led the pack with overall averages of 59.56% and 59.57%, respectively.
  • Open-Source Leaders: Qwen2.5-VL-7B was the strongest open-source model at 55.86%, significantly outperforming others like LLaVA-OneVision-7B (38.36%).
  • Prompting Gains: For GPT-4o, visual prompting (VP-grid) provided a ~10% performance boost by helping the model disambiguate image order and perform precise spatial matching. However, for Qwen2.5-VL, the same prompt caused a ~20% performance drop.
Performance gains and losses from different prompting methods

Limitations and Future Work

The authors acknowledge that VLM2-Bench, while diverse, does not cover every possible visual scenario. The scale is currently limited to ~3,000 cases, and the automated evaluation for open-ended questions (using GPT-4o as a judge) may have its own biases.

Future work includes expanding task diversity, moving toward multi-turn conversation evaluations, and exploring "self-play" or synthetic data pretraining to help models better learn the "grammar" of visual relationships without relying solely on language-aligned data.

Summary updated

Abstract

Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. Despite the extensive knowledge that vision-language models (VLMs) possess, it remains largely unexplored whether they are capable of performing this fundamental task. To address this, we introduce \textbf{VLM2-Bench}, a benchmark designed to assess whether VLMs can Visually Link Matching cues, with 9 subtasks and over 3,000 test cases. Comprehensive evaluation across twelve VLMs, along with further analysis of various language-side and vision-side prompting methods, leads to a total of eight key findings. We identify critical challenges in models' ability to link visual cues, highlighting a significant performance gap. Based on these insights, we advocate for (i) enhancing core visual capabilities to improve adaptability and reduce reliance on prior knowledge, (ii) establishing clearer principles for integrating language-based reasoning in vision-centric tasks to prevent unnecessary biases, and (iii) shifting vision-text training paradigms toward fostering models' ability to independently structure and infer relationships among visual cues.
BibTeX
@inproceedings{zhang2025vlm,
  author = {Jianshu Zhang and Dongyu Yao and Renjie Pi and Paul Pu Liang and Yi R. Fung},
  title = {{VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure