When we ask a vision-language model to describe an image, how do we know if the generated description is accurate? Traditionally, the NLP and computer vision communities have relied on reference-based metrics like BLEU, ROUGE, or CIDEr. These metrics compare the generated text against a small set of human-written "gold standard" captions.
However, this approach introduces a fundamental bottleneck. As noted in the comprehensive survey From Show to Tell: A Survey on Deep Learning-Based Image Captioning, text-to-text metrics are poor proxies for human judgment because they fail to measure the direct semantic alignment between an image and its description. An image contains a near-infinite number of valid descriptions depending on what details a speaker chooses to emphasize. If a model generates a perfectly accurate description that happens to use different vocabulary or focus on a different visual element than the human reference, traditional metrics penalize it. Furthermore, as Marcella Cornia et al. point out in Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives, modern large models generate highly detailed descriptions that suffer from a "stylistic mismatch" when graded against brief human annotations.
To solve this, researchers have shifted toward reference-free multimodal evaluation. Instead of comparing text to text, these metrics eliminate the human reference entirely and evaluate the generated text by measuring its alignment directly against the source visual pixels.
Shared Embedding Spaces
The simplest way to evaluate a caption without a reference is to project both the image and the candidate text into a shared mathematical space where distance correlates with semantic similarity.
[ Image ] --------> [ Vision Encoder ] -------\
> ---> [ Cosine Similarity Score ]
[ Candidate Text ] -> [ Text Encoder ] -------/
Jack Hessel et al. formalized this intuition with CLIPScore: A Reference-free Evaluation Metric for Image Captioning. By utilizing a pre-trained contrastive vision-language model (CLIP), CLIPScore calculates the cosine similarity between the image vector and the text vector. This approach allows for automated evaluation on unlabelled datasets where human references are entirely unavailable.
Subsequent work has refined how these embeddings are trained and applied:
- Contrastive Fine-Tuning: Hwanhee Lee et al. introduced UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning, which fine-tunes a vision-language model on synthetic negative samples (such as captions with swapped words or incorrect objects) to make the embedding space highly sensitive to descriptive errors.
- Cross-Modal Generation: For evaluating text-to-image consistency in generative models, Tianwei Zhou et al. demonstrate in Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment that fusing multi-scale image features with prompt embeddings provides a robust metric for semantic consistency.
- Video Granularity: This paradigm scales to video as well. Yaya Shi et al. developed EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching, which computes alignment at both a coarse level (the entire video vs. the entire sentence) and a fine level (individual video frames vs. specific words) to catch temporal hallucinations.
Granular Fidelity and Object Verification
While global embedding metrics are cheap and fast, they suffer from a major flaw: they look at the "average" meaning of an image and can easily miss fine-grained errors, such as a model hallucinating a small object that isn't there. To fix this, a subset of reference-free metrics breaks the image down into discrete visual entities.
Early work by Pranava Madhyastha et al. in VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions bypassed reference text by running an object detector over the source image and comparing the generated caption directly to the resulting list of detected object labels.
Modern iterations make this grounding much more sophisticated. Marcella Cornia et al. proposed BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues, which maps dense, regional visual features directly into the evaluation space, ensuring that the metric explicitly penalizes object hallucinations. For long-form descriptions where models are prone to drifting, Monika Wysoczańska et al. introduced OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models. OVFact measures precision as the exact ratio of textually mentioned entities that can be successfully grounded back to an open-vocabulary concept within the image pixels.
Large Multimodal Models as Judges
With the advent of advanced Large Multimodal Models (LMMs), evaluation has evolved from simple vector math to active semantic reasoning. Instead of calculating a distance score, we can ask an intelligent model to look at the image and fact-check the caption.
An early conceptual step toward this was taken by Hwanhee Lee et al. in QACE: Asking Questions to Evaluate an Image Caption. QACE automatically generates questions based on the candidate caption and uses a question-answering system to verify the answers directly against the image pixels, treating the visual input as the ultimate source of truth.
Today, models like GPT-4o or specialized open-source evaluators act as direct judges. Seongyun Lee developed Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation, which evaluates an image, an instruction, and a model's response simultaneously. This direct triplet evaluation prevents the information bottleneck that occurs when visual data is compressed into a text-only proxy before grading.
[ Image ] + [ Instruction ] + [ Candidate Response ]
|
v
[ Multimodal Model Judge ]
|
v
[ Score + Natural Language Explanation ]
Metrics like FLEUR (developed by Yebin Lee et al.) and EXPERT (developed by Hyunjong Kim et al.) leverage this paradigm to provide not just a numerical score, but a structured, natural language explanation of why a caption succeeded or failed. Similarly, Tony Cheng Tong et al. introduced G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o, which uses Chain-of-Thought reasoning and probabilistic outputs to achieve high alignment with human judgment.
This reasoning capability is particularly vital in complex domains. For instance, Heng Ji et al. demonstrate in Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning that a visual entailment model can successfully verify whether mathematical and factual claims in a chart caption are logically supported by the chart itself.
Even when a judge model lacks direct visual perception capabilities, reference-free evaluation can still function via text proxies. As shown by Anas Awadalla et al. in VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models, providing a text-only LLM with a dense, highly specific textual description of the image allows it to act as an effective proxy judge for multimodal outputs.
Generative Loops and Information Bottlenecks
Some of the most creative reference-free frameworks evaluate alignment by closing the loop between generation and reconstruction, or by measuring how much visual context reduces a model's internal uncertainty.
Round-Trip Reconstruction
If a caption perfectly describes an image, a separate model should be able to reconstruct that image using only the caption. JIA-HONG HUANG et al. proposed this exact "round-trip" mechanism in A Novel Evaluation Framework for Image2Text Generation. The framework takes a generated caption, passes it to a text-to-image model to synthesize a new image, and then measures the structural similarity between the original image and the reconstructed image.
[ Original Image ] ---> ( Captioning Model ) ---> [ Generated Caption ]
| |
| (Compare Similarity) v
|<--------------------------------------- ( Text-to-Image Model )
Lele Cao et al. expanded on this concept with Introducing GenCeption for Multimodal LLM Benchmarking: You May Bypass Annotations, creating an iterative "visual telephone" loop to track semantic drift and hallucination rates without any human labels. A related approach is used by Yi Gui et al. in LaTCoder: Converting Webpage Design to Code with Layout-as-Thought, where generated front-end code layouts are verified by combining pixel-level reconstruction errors with vision-language embedding scores.
Information-Theoretic Grounding
Another way to verify alignment is to measure how much a model relies on the image versus its internal language priors. Ji Ma addresses this in C3L: Content Correlated Vision-Language Instruction Tuning Data Generation via Contrastive Learning by measuring the divergence between a model's output probability distributions when it is shown the image versus when the image is withheld. If the distribution barely changes, the model is simply guessing based on text priors rather than grounding its answer in the visual data.
Similarly, Shenghui Chen et al. introduced VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR. VIBE uses pointwise mutual information to calculate a grounding metric based on how much a source video improves a model's ability to reconstruct masked portions of a generated summary text, providing a mathematical measure of utility without human references.
Vulnerabilities and Robustness
Despite their rapid advancement, reference-free metrics are not infallible. Because they rely heavily on pre-trained multimodal embeddings or language priors, they can exhibit blind spots that humans easily spot.
Saba Ahmadi et al. exposed these vulnerabilities in An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics. They discovered that metrics like CLIPScore fail nearly 46% of the time when presented with "hard" incorrect captions—captions that contain high lexical overlap with the truth but introduce critical logical or grammatical errors. Furthermore, Elisa Kreiss et al. demonstrated in ContextRef: Evaluating Referenceless Metrics for Image Description Generation that referenceless metrics often fail to account for the broader context (such as an accompanying news article) that defines whether a description is actually relevant and appropriate.
To mitigate these flaws, researchers are developing perturbation-robust variants. For example, Yongil Kim et al. introduced PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning, which explicitly trains encoders to recognize and penalize lexical noise, word repetitions, and jumbled grammar across multiple languages.
By shifting the ground truth from static human text to dynamic cross-modal verification, reference-free evaluation provides a scalable, cheap, and linguistically flexible paradigm that matches the open-ended nature of modern vision-language models.
Key Foundational Work
UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning (2021) — contributed 2 concept elements to this direction.
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions (2019) — contributed 2 concept elements to this direction.
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021) — contributed 1 concept elements to this direction.