When we evaluate how well an AI model describes an image or answers a question about a video, our instinct is to check its output against a human-written answer. For years, the field relied on text-only metrics like BLEU, ROUGE, or BERTScore to do exactly this. If the model’s text closely matched the human reference text, it received a high score.
However, this approach contains a fundamental blind spot: the evaluation metric never actually looks at the image.
By treating multimodal evaluation as a pure text-matching problem, traditional metrics create a profound disconnect between automated scores and human perception, as noted by Raffaella Bernardi et al. and Marcella Cornia et al.. To truly evaluate a vision-language model, the evaluation metric itself must be visually grounded.
The Failure Modes of Text-Only Evaluation
To understand why text-only metrics fail, consider two distinct problems: hallucination blindness and the lexical penalty.
1. Hallucination Blindness
Imagine an image showing a black cat on a red mat. A human reference caption reads: "A black cat sitting on a mat." If a model generates the caption "A black dog sitting on a mat," a text-only metric will award it an exceptionally high score. The sentence structure is identical, and in language-only embedding spaces, "cat" and "dog" are highly similar concepts.
Because the metric is blind to the source pixels, it cannot detect that the model has hallucinated an entirely wrong animal. As highlighted by Jack Hessel et al. and the authors of EMScore, text-only metrics routinely pass plausible-sounding but visually fraudulent descriptions because they measure linguistic similarity rather than factual alignment with the visual evidence.
2. The Lexical Penalty
Now imagine the same image, but the model generates: "A dark feline resting on a crimson rug." Visually, this is an exceptionally accurate and descriptive caption. Lexically, however, it shares almost no words with the human reference.
Rigid n-gram metrics or even standard text embeddings will penalize this caption heavily. Huishan Ji et al. and Neelabh Sinha et al. demonstrate that this rigid reference-matching penalizes perfectly valid, creative, or structurally varied responses in tasks like image captioning and Visual Question Answering (VQA). Pierre Dognin et al. similarly argue that models optimizing for true semantic alignment are unfairly suppressed by legacy text-matching metrics.
Text-Only Evaluation (Blind to Image):
[Generated Text] <---> [Human Reference Text] --> High similarity, but ignores visual errors
Visually Grounded Evaluation:
[Generated Text] <---> [Source Image/Video] --> Verifies physical and semantic truth
Bringing Pixels into the Metric
To bridge this gap, researchers have developed metrics that actively incorporate the visual source into the evaluation loop. They do this through three primary mechanisms: cross-modal spaces, image-conditioned text representations, and vision-language judges.
Reference-Free Cross-Modal Alignment
The most direct way to eliminate the limitations of human reference texts is to get rid of them entirely. Instead of measuring text-to-text similarity, reference-free metrics project both the visual source and the generated text into a shared embedding space.
Jack Hessel et al. introduced CLIPScore, which uses a joint vision-language model (CLIP) to calculate the direct cosine similarity between an image embedding and a candidate caption embedding. Because the text encoder is trained specifically to align with visual concepts, the metric naturally penalizes words that do not correspond to the pixels.
The authors of EMScore extended this intuition to video captioning, matching video frames and words at both coarse (video-to-sentence) and fine (frame-to-word) granularities to ensure that the temporal flow of the text matches the visual progression of the video.
Image-Conditioned Text Representations
If human references are available, they can still be valuable—provided they are interpreted through the lens of the image. Standard language models generate embeddings based purely on textual context. In contrast, multimodal metrics condition text representations on the visual features of the image.
Hwanhee Lee et al. developed ViLBERTScore, which passes both the image regions and the text captions through a multi-modal transformer. This allows the text embeddings to change dynamically based on what is visible. If a model describes a specific attribute of an object, the metric can verify whether that attribute is visually supported by the corresponding image region.
Taking structure a step further, Sijin Wang et al. introduced FAIEr, which constructs a unified scene graph blending image features and reference text to explicitly measure fidelity (visual accuracy) and adequacy (whether the text covers the most important visual details).
Vision-Language Models as Judges
With the advent of powerful Multimodal Large Language Models (MLLMs), evaluation has evolved from calculating distance vectors to leveraging deep contextual reasoning. Instead of relying on rigid mathematical formulas, these approaches use advanced models to read the image and judge the text according to explicit rubrics.
Seongyun Lee et al. introduced PROMETHEUS-VISION, an open-source VLM evaluator trained on thousands of fine-grained score rubrics. It acts as a flexible judge, analyzing the image alongside the generated text to provide human-like, interpretable feedback.
Similarly, the GoEval framework introduced by Neelabh Sinha et al. uses state-of-the-art MLLMs to evaluate generative VQA responses, ensuring that answers are rewarded for being factually grounded in the visual scene, regardless of their exact phrasing.
Why Visually Grounded Metrics Work
The transition from text-only metrics to visually grounded evaluators works because it respects the underlying nature of multimodal representations. As Zhihong Chen et al. demonstrated, text encoders trained purely on language (like BERT) develop a fundamentally different understanding of similarity than those trained on image-text pairs (like CLIP). Language-only models excel at identifying paraphrases, but they are blind to whether two sentences describe the same physical reality.
By anchoring evaluation metrics in the visual domain, the AI community can now:
- Verify physical correspondence: Ensure that generated text accurately reflects the specific visual features and layout of the scene (Prateek Chhikara et al.).
- Maintain visual coherence: Track consistency across complex, multi-step visual sequences without losing subtle nuances through intermediate text conversions (Xiaoxin Lu et al.).
- Understand communicative intent: Evaluate whether a description captures the underlying logic, structure, and discourse relations of an image, rather than just matching keywords (Mert Inan et al.).
Ultimately, visually grounded metrics close the evaluation gap by ensuring that models are judged not just on their ability to write fluent prose, but on their capacity to truly see.
Key Foundational Work
Re-evaluating Automatic Metrics for Image Captioning (2017) — contributed 1 concept elements to this direction.
Learning to Evaluate Image Captioning (2018) — contributed 1 concept elements to this direction.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract) (2017) — contributed 1 concept elements to this direction.