Abstract
Evaluating the relevance of images to text queries typically requires extensive human annotation, creating a bottleneck for large-scale Information Retrieval (IR) tasks. This paper investigates the use of Vision-Language Models (VLMs), such as GPT-4V and LLaVA, as automated "proxy" judges for relevance assessment. Evaluating these models on the TREC-AToMiC 2023 dataset, the authors find that instruction-tuned VLMs significantly outperform traditional metrics like CLIPScore in matching human judgments. However, the study also reveals an "evaluation bias," where model-based judges tend to favor retrieval systems that share their underlying architecture.
Introduction
In Information Retrieval, the gold standard for system evaluation is the "Cranfield-style" test collection, which relies on human-annotated relevance labels. As retrieval tasks evolve toward complex multimedia contexts—such as finding illustrative images for specific Wikipedia sections—the cost and time required for human labeling become prohibitive. While automated metrics like CLIPScore offer a scalable alternative, they often lack the reasoning capabilities required to distinguish between truly illustrative content and merely decorative imagery. This research explores whether modern, instruction-tuned VLMs can bridge this gap by acting as zero-shot relevance judges.
Methodology
The researchers treated relevance judgment as a zero-shot task, utilizing prompt engineering rather than model fine-tuning. They compared a baseline (CLIP) against open-source models (LLaVA-7b, LLaVA-13b) and a state-of-the-art closed-source model (GPT-4V).
As shown in the prompt template, the models were provided with rich contextual information, including page titles, section titles, and full section text. The instructions explicitly directed the models to evaluate if an image is a significant illustrative aid or merely decorative. To align model outputs with human labels (graded 0 to 2), the authors applied a quantile-based heuristic to map the models' 1–100 relevance scores into discrete categories.
Experimental Results
The study utilized the TREC-AToMiC 2023 test collection, consisting of nearly 10,000 image-text pairs. The primary finding was that instruction-tuned models are far more effective at capturing human-like reasoning than embedding-based similarity metrics.
Ranking Correlation and Agreement
LLaVA and GPT-4V achieved a Kendall’s $\tau$ of approximately 0.4 for NDCG@10, effectively doubling the performance of the CLIPScore baseline ($\tau \approx 0.2$). Furthermore, GPT-4V showed "slight agreement" with humans (Cohen’s $\kappa = 0.08$), whereas CLIPScore actually exhibited a negative agreement score, suggesting its judgments were less aligned with human assessors than random chance in certain contexts.
The Evaluation Bias Trap
A critical insight from the study is the presence of "evaluation bias." The authors observed that model-based judges tend to favor retrieval systems that are architecturally similar to themselves. For instance, CLIP-based judges were over 100% biased in favor of CLIP-based retrieval systems. Even LLaVA and GPT-4V showed a slight preference for CLIP-based systems, likely because they utilize CLIP as their visual backbone. This suggests that while VLMs are powerful judges, their evaluations may not be entirely neutral across different retrieval architectures.
Distribution Alignment
Analysis of score distributions revealed that GPT-4V’s judgments closely mirrored the cumulative distribution of human labels. In contrast, LLaVA models tended to be "over-optimistic," frequently labeling non-relevant items as "Related" and failing to identify as many "Non-relevant" cases as human assessors.
Conclusion and Future Work
While instruction-tuned VLMs represent a significant step toward automated IR evaluation, challenges remain. The cost of using proprietary models like GPT-4V is still significant for large-scale datasets. Additionally, the identified architectural bias must be addressed to ensure fair evaluation across diverse retrieval systems. Future research should explore "listwise" evaluation, where models compare multiple images simultaneously, and investigate methods to de-bias model-based judgments.
Summary updated