Introduction
In the field of Information Retrieval (IR), the "Cranfield-style" test collection—comprising queries, a document corpus, and human-generated relevance judgments (qrels)—remains the gold standard for evaluation. However, as datasets expand to include millions of images and complex, long-form text, human annotation has become a significant bottleneck. It is inherently slow, expensive, and difficult to scale.
While automated metrics like CLIPScore are widely used, they are often too coarse to capture fine-grained details or handle the long-context text typical of real-world applications like Wikipedia. Researchers from the University of Waterloo investigated whether the latest generation of Vision-Language Models (VLMs) could serve as effective "proxy" human assessors to facilitate faster and more scalable evaluation.
Methodology
The study proposes a fully automatic evaluation paradigm utilizing three distinct types of VLMs:
- Embedding-based: CLIP (specifically CLIPScore).
- Open-source LLM-based: LLaVA (7b and 13b versions).
- Closed-source LLM-based: GPT-4V.
The core of this approach lies in Prompt Engineering. The researchers designed a template to provide the VLM with the necessary context and criteria to mimic a human assessor.
As shown in the prompt design, models are provided with page titles, section descriptions, and specific relevance criteria (e.g., ensuring images are significant rather than merely decorative). To compare model outputs with human "ground truth," the resulting 1-100 scores were mapped into three graded levels (0: Non-relevant, 1: Related, 2: Relevant) using a quantile-based heuristic.
Experimental Setup
The study utilized the TREC-AToMiC 2023 test collection, focusing on the task of image suggestion—identifying the best image to illustrate a specific section of a Wikipedia article. The dataset included 9,818 image-text pairs across 74 test topics. Model-generated judgments were benchmarked against official relevance judgments provided by NIST (National Institute of Standards and Technology) assessors.
Results and Analysis
Instruction Tuning and Performance
A key finding is that instruction tuning is essential for effective judgment. Unlike basic CLIP, which relies on simple similarity, models like LLaVA and GPT-4V can reason about task instructions. This capability allows them to process complex, long-form text that exceeds CLIP’s 77-token limit. Consequently, LLaVA and GPT-4V achieved a Kendall’s $\tau$ of approximately 0.4, doubling the performance of CLIPScore (~0.2) in terms of ranking correlation with human assessors.
Distributional Alignment
When analyzing the Cumulative Distribution Function (CDF) of the scores, GPT-4V demonstrated a unique ability to mimic human judgment patterns. While other models showed significant deviations, GPT-4V's internal scoring distribution aligned much more closely with the human "curve."
Evaluation Bias and Agreement
Despite their strengths, the study identified an "evaluation bias" phenomenon. Model-based evaluators tend to favor retrieval systems that share their underlying architecture. Since LLaVA and GPT-4V often utilize CLIP-based visual encoders, they showed a bias toward CLIP-based retrieval systems, although this bias was significantly lower than that of CLIPScore itself (Relative $\Delta$ reduced from 114.7 to ~55-64).
In terms of direct agreement, GPT-4V achieved a Cohen’s $\kappa$ of 0.080. While categorized as "slight" agreement, this represents a major improvement over CLIPScore, which exhibited a negative correlation (-0.096) with human agreement.
Conclusion and Future Work
Vision-Language Models offer a promising, cost-effective alternative to human annotation; the researchers noted that the GPT-4V experiments cost roughly USD 150, a fraction of the cost of professional NIST assessors. However, several challenges remain. The current "pointwise" estimation (evaluating one pair at a time) could be evolved into "listwise" ranking for more nuanced comparisons. Furthermore, the gap between model logic and human intuition (as evidenced by the low $\kappa$ score) suggests that further prompt optimization and research into mitigating architectural bias are necessary before VLMs can fully replace human assessors in IR benchmarking.
Summary updated