Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation

Jheng-Hong Yang, Jimmy Lin

Introduction

In the field of Information Retrieval (IR), the "Cranfield-style" test collection—comprising queries, a document corpus, and human-generated relevance judgments (qrels)—remains the gold standard for evaluation. However, as datasets expand to include millions of images and complex, long-form text, human annotation has become a significant bottleneck. It is inherently slow, expensive, and difficult to scale.

While automated metrics like CLIPScore are widely used, they are often too coarse to capture fine-grained details or handle the long-context text typical of real-world applications like Wikipedia. Researchers from the University of Waterloo investigated whether the latest generation of Vision-Language Models (VLMs) could serve as effective "proxy" human assessors to facilitate faster and more scalable evaluation.

Methodology

The study proposes a fully automatic evaluation paradigm utilizing three distinct types of VLMs:

  1. Embedding-based: CLIP (specifically CLIPScore).
  2. Open-source LLM-based: LLaVA (7b and 13b versions).
  3. Closed-source LLM-based: GPT-4V.

The core of this approach lies in Prompt Engineering. The researchers designed a template to provide the VLM with the necessary context and criteria to mimic a human assessor.

The prompt template used to provide VLMs with context, relevance instructions, and output formatting for automatic assessment.

As shown in the prompt design, models are provided with page titles, section descriptions, and specific relevance criteria (e.g., ensuring images are significant rather than merely decorative). To compare model outputs with human "ground truth," the resulting 1-100 scores were mapped into three graded levels (0: Non-relevant, 1: Related, 2: Relevant) using a quantile-based heuristic.

Experimental Setup

The study utilized the TREC-AToMiC 2023 test collection, focusing on the task of image suggestion—identifying the best image to illustrate a specific section of a Wikipedia article. The dataset included 9,818 image-text pairs across 74 test topics. Model-generated judgments were benchmarked against official relevance judgments provided by NIST (National Institute of Standards and Technology) assessors.

Results and Analysis

Instruction Tuning and Performance

A key finding is that instruction tuning is essential for effective judgment. Unlike basic CLIP, which relies on simple similarity, models like LLaVA and GPT-4V can reason about task instructions. This capability allows them to process complex, long-form text that exceeds CLIP’s 77-token limit. Consequently, LLaVA and GPT-4V achieved a Kendall’s $\tau$ of approximately 0.4, doubling the performance of CLIPScore (~0.2) in terms of ranking correlation with human assessors.

Distributional Alignment

When analyzing the Cumulative Distribution Function (CDF) of the scores, GPT-4V demonstrated a unique ability to mimic human judgment patterns. While other models showed significant deviations, GPT-4V's internal scoring distribution aligned much more closely with the human "curve."

Cumulative Distribution Function (CDF) of scores, illustrating how GPT-4V's scoring distribution closely follows the human judgment pattern compared to CLIP and LLaVA.

Evaluation Bias and Agreement

Despite their strengths, the study identified an "evaluation bias" phenomenon. Model-based evaluators tend to favor retrieval systems that share their underlying architecture. Since LLaVA and GPT-4V often utilize CLIP-based visual encoders, they showed a bias toward CLIP-based retrieval systems, although this bias was significantly lower than that of CLIPScore itself (Relative $\Delta$ reduced from 114.7 to ~55-64).

In terms of direct agreement, GPT-4V achieved a Cohen’s $\kappa$ of 0.080. While categorized as "slight" agreement, this represents a major improvement over CLIPScore, which exhibited a negative correlation (-0.096) with human agreement.

Conclusion and Future Work

Vision-Language Models offer a promising, cost-effective alternative to human annotation; the researchers noted that the GPT-4V experiments cost roughly USD 150, a fraction of the cost of professional NIST assessors. However, several challenges remain. The current "pointwise" estimation (evaluating one pair at a time) could be evolved into "listwise" ranking for more nuanced comparisons. Furthermore, the gap between model logic and human intuition (as evidenced by the low $\kappa$ score) suggests that further prompt optimization and research into mitigating architectural bias are necessary before VLMs can fully replace human assessors in IR benchmarking.

Summary updated

Abstract

Vision--Language Models (VLMs) have demonstrated success across diverse applications, yet their potential to assist in relevance judgments remains uncertain. This paper assesses the relevance estimation capabilities of VLMs, including CLIP, LLaVA, and GPT-4V, within a large-scale \textit{ad hoc} retrieval task tailored for multimedia content creation in a zero-shot fashion. Preliminary experiments reveal the following: (1) Both LLaVA and GPT-4V, encompassing open-source and closed-source visual-instruction-tuned Large Language Models (LLMs), achieve notable Kendall's $τ\sim 0.4$ when compared to human relevance judgments, surpassing the CLIPScore metric. (2) While CLIPScore is strongly preferred, LLMs are less biased towards CLIP-based retrieval systems. (3) GPT-4V's score distribution aligns more closely with human judgments than other models, achieving a Cohen's $κ$ value of around 0.08, which outperforms CLIPScore at approximately -0.096. These findings underscore the potential of LLM-powered VLMs in enhancing relevance judgments.
BibTeX
@inproceedings{yang2024toward,
  author = {Jheng-Hong Yang and Jimmy Lin},
  title = {{Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation}},
  booktitle = {LLM4Eval @ SIGIR},
  year = {2024},
  url = {https://openreview.net/forum?id=BdnSbNdQHA},
}
Figure