PURE: A Diagnostic Benchmark for Multi-Granularity Multimodal Identity Matching

Hengyu You, Xintong Wang, Jin Cao, Liang DingarXiv 2026 · arXiv

TL;DR

The PURE benchmark introduces a diagnostic framework to evaluate how Multimodal Large Language Models (MLLMs) handle "identity matching"—determining if two records represent the same object. By breaking down performance into Perception, Understanding, Reasoning, and Judgment across different granularities, it reveals that while models excel at extracting attributes, they frequently stumble when localizing visual differences or applying complex matching rules.

What Problem Does This Paper Address?

In the world of e-commerce, "sameness" is a moving target. Are two listings for a "pointed-toe flat" the same? At a base level, yes—they share the same design. But at a variant level, a beige size 37 shoe is fundamentally different from a brown size 37 shoe.

Current identity matching systems often provide a single "match/no-match" score. This aggregate metric is a "black box" that hides specific failure modes. A model might fail because it didn't "see" a kitten heel (Perception), misunderstood the word "beige" (Understanding), failed to notice a conflict between the image and the title (Reasoning), or simply didn't know that a color difference should trigger a "non-match" at the variant level (Judgment). The authors propose PURE to diagnose these specific capability gaps.

Figure 1: Multi-granularity identity matching and the PURE task taxonomy. (a) Records A and B differ in heel structure and do not match at the base level (Case 1). Within a shared base design, records that differ in color (Case 2) or heel height (Case 3) do not match at the variant level. (b) PURE organizes 14 tasks by diagnostic dimension and identity-matching granularity.

Methods

The researchers developed an error-driven construction pipeline to ensure the benchmark is genuinely challenging. Instead of random sampling, they specifically targeted "hard" cases where existing state-of-the-art models (both embedding-based matchers and generative MLLMs) failed.

  1. Data Collection: They identified real-world record pairs misclassified by current systems.
  2. Task Assignment: Using Gemini 3.1 Pro, they analyzed why a pair was difficult and assigned it to one of 14 diagnostic tasks (e.g., "Visual Difference Localization" or "Attribute Expression Equivalence").
  3. Question Generation: They used MLLMs to generate multiple-choice questions conditioned on the specific task and identity criteria.
  4. Quality Assurance: A rigorous multi-stage process involving multi-model screening and final verification by two human domain experts.
Figure 2: Construction pipeline of PURE. ❶Real-world base-level and variant-level record pairs jointly misclas- sified by an embedding matcher and a generative MLLM form the source pool. ❷Identity criteria and the task taxonomy guide evidence-grounded analysis and task assignment. ❸Gemini 3.1 Pro generates task-conditioned questions. ❹Candidates undergo multi-model quality screening followed by expert review.

Datasets

The paper introduces the PURE benchmark, which consists of:

  • 1,457 multiple-choice questions across 14 tasks.
  • Four capability dimensions: Perception, Understanding, Reasoning, and Evidence-grounded Judgment.
  • Real-world e-commerce data: Sourced from Alibaba Group, featuring diverse photography, titles, and attribute expressions.

The authors compared PURE against existing benchmarks like M4Bench (visual comparison) and FGVQA (fine-grained recognition), finding strong correlations but noting that PURE is unique in its focus on multi-granularity identity rules.

Key Insights and Ideas

  • The "Understanding" vs. "Judgment" Gap: Models are surprisingly good at extracting information (e.g., reading text from an image or normalizing attribute names) but struggle significantly more with applying that information to make a final matching decision.
  • Blind Faith in Text: Modality ablation studies showed that models often over-rely on textual descriptions. In some cases, removing the product title actually increased accuracy because the model stopped hallucinating conflicts based on non-discriminative wording differences (e.g., "lightweight jacket" vs. "hooded jacket" for the same item).
  • Recall Asymmetry: At the base level, models are "optimistic"—they are better at recognizing matches than non-matches. At the variant level, the reverse is true; they become "pessimistic" and are more reliable at identifying non-matching pairs.
  • Thinking Boosts Reasoning: Enabling "thinking" (Chain-of-Thought) led to massive gains (up to 13.8 percentage points), particularly in tasks requiring visual localization and complex reasoning.
Figure 3: Modality-ablation results. Removing titles (red) sometimes improves accuracy in conflict detection (C2) and rule judgment (D1), suggesting models can be misled by textual noise.

Key Results

The authors evaluated ten recent MLLMs, including Gemini 3.8 Flash, GPT-5.6 Sol, and various Qwen models.

  • Top Performer: Gemini 3.8 Flash led the pack with an overall score of 86.0%.
  • Dimension Scores: "Understanding" tasks saw the highest mean scores (~89%), while "Judgment" was the lowest (~61%).
  • Diagnostic Relevance: PURE scores showed a high Spearman correlation (ρ = 0.952) with direct identity matching accuracy, proving that these diagnostic tasks are indeed the building blocks of the final matching decision.

Limitations and Future Work

  • Explicit vs. Implicit Matching: PURE evaluates explicit multiple-choice responses. It does not directly characterize the internal representations used in embedding-based retrieval systems.
  • Domain Specificity: The benchmark is currently focused on e-commerce. While this is a rich domain, the authors acknowledge that identity matching in other fields (like medical imaging or legal document verification) may require different rules and tasks.
  • Inherited Biases: Because the source pool relies on errors from existing models, the benchmark might inherit some of their coverage biases.

Future work could involve expanding PURE to more languages and product categories, or developing new training objectives specifically designed to bridge the "Understanding-to-Judgment" gap.

Summary updated

Abstract

Identity matching asks whether two records denote the same identity, yet the answer depends on the target granularity. Two records may match at the base level but not at the variant level: they share an underlying design while differing in attributes such as color or size. Final matching accuracy does not separately characterize the capabilities involved. To study this problem, we use e-commerce as a challenging real-world domain. We introduce PURE, a diagnostic benchmark for multi-granularity multimodal identity matching covering four capability dimensions: Perception, Understanding, Reasoning, and Evidence-grounded Judgment. The benchmark comprises 1,457 multiple-choice questions across 14 tasks, derived from real-world matching pairs and verified by two domain experts. Across ten recent MLLMs, overall scores range from 47.1% to 86.0%. Understanding receives the highest mean score and Judgment the lowest, with weaknesses in visual difference localization and variant-level rule judgment. Modality ablations show that models benefit from multimodal evidence but can incorrectly identify conflicts between matching records. In direct matching, most models recognize matching pairs more reliably than non-matching pairs at the base level, whereas the reverse holds at the variant level. PURE therefore reveals what aggregate matching accuracy hides: not only whether models judge a pair correctly, but where they struggle to interpret evidence and apply matching criteria.
BibTeX
@misc{you2026pure,
  author = {Hengyu You and Xintong Wang and Jin Cao and Liang Ding},
  title = {{PURE: A Diagnostic Benchmark for Multi-Granularity Multimodal Identity Matching}},
  year = {2026},
  eprint = {2611.22343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22343},
}
Figure