When we evaluate how well a machine describes an image, we are measuring its ability to translate visual geometry into natural language. For years, the standard tools for this task were metrics borrowed from machine translation and text summarization, such as BLEU, ROUGE, and CIDEr. These metrics operate on a simple assumption: the quality of a generated caption is proportional to its lexical overlap with a human-written reference sentence.
However, evaluating language grounded in vision introduces a unique challenge. While a text document has a relatively constrained set of translations, a single visual scene can be accurately described in an infinite number of ways. By relying strictly on exact word matching, traditional metrics create a double failure mode: they penalize valid creative diversity while remaining blind to catastrophic factual errors.
The Failure Modes of Lexical Matching
To understand why word-matching fails, consider how humans judge a description versus how an n-gram metric scores it. As noted by Oriol Vinyals et al. in their foundational work Show and tell: A neural image caption generator, n-gram metrics often fail to account for semantic correctness or descriptive creativity. A model can produce a beautifully accurate description that happens to use synonyms or an alternative syntactic structure, only to be heavily penalized by an automated score.
Micah Hodosh et al. demonstrated in Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Extended Abstract) that word-overlap metrics correlate very weakly with human expert assessments. This creates a "reference bias" or a metric trap: it forces models to mimic the exact vocabulary of the training references rather than focusing on visual fidelity, a limitation thoroughly detailed by Sijin Wang et al. in FAIEr: Fidelity and Adequacy Ensured Image Caption Evaluation.
Worse yet is the problem of structural blindness. N-gram metrics count the presence of words but struggle to verify how those words are bound together. If a reference caption is "A dog chases a cat," a model that outputs "A cat chases a dog" shares 100% of its vocabulary and n-grams, yet describes an entirely different physical reality. As Fan Lu et al. point out in Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning, this structural blindness allows models to receive high scores for descriptions that contain correct keywords but completely misinterpret the relationships between objects.
This vulnerability becomes critical when dealing with "hard negatives"—captions that are linguistically similar to the truth but factually incorrect. In An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics, Saba Ahmadi et al. revealed that even advanced metrics fail to distinguish correct captions from hard negatives nearly 46% of the time. Similarly, Brian Gordon et al. argue in Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline that standard metrics fail to penalize explicit factual contradictions if the surface-level vocabulary matches closely.
Grounding Evaluation in Propositional Logic
To fix this, the field shifted from evaluating strings to evaluating propositions. The core intuition is that an image description is not just a sequence of tokens; it is a structured claim about the objects, attributes, and relations present in a scene.
Surface Text: "A brown dog chases a cat"
│
▼ (Semantic Parsing)
Scene Graph:
[Dog] ──(Attribute)──> [Brown]
│
(Relation: Chasing)
│
▼
[Cat]
Peter Anderson et al. pioneered this direction with SPICE: Semantic Propositional Image Caption Evaluation. Instead of comparing sentences directly, SPICE uses a semantic parser to convert both the candidate and reference captions into scene graphs. A scene graph abstracts away the surface syntax, mapping the text into discrete semantic propositions: objects ($O$), attributes ($A$), and relations ($R$).
By calculating the precision and recall over these graph tuples rather than word n-grams, SPICE evaluates whether the model correctly identified that a dog is brown and that the dog is chasing the cat. Whether the model writes "A brown dog chases a cat" or "A cat is pursued by a dog that is brown," the underlying scene graph remains identical, eliminating the penalty for syntactic variation.
A parallel approach was developed by Lily D. Ellebracht et al. in Semantic Tuples for Evaluation of Image to Sentence Generation, which introduced BAST (Bag of Aggregated Semantic Tuples). BAST explicitly breaks sentences down into "who did what where" components, ensuring that the evaluation prioritizes structural semantic components over simple lexical similarity. In his comprehensive taxonomy of the field, Desmond Elliott notes in Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract) that frameworks utilizing these structural representations consistently align more closely with human judgment of descriptive quality than traditional translation metrics.
Continuous Semantics and Diagnostic Dimensions
While scene graphs solve the structural binding problem, they can still be sensitive to the brittle choices of a parser. To introduce more flexibility, researchers explored continuous semantic spaces and multi-dimensional diagnostic frameworks.
Instead of requiring exact token matches, Aykut Erdem et al. demonstrated in Re-evaluating Automatic Metrics for Image Captioning that Word Mover’s Distance (WMD) is highly effective for caption evaluation. WMD treats sentences as distributions of word embeddings and measures the minimum "traveling cost" to transform one sentence into another. This allows the metric to naturally recognize synonyms and semantic equivalence without explicit graph parsing. To capture even broader context, Chao Zeng et al. introduced Contrastive semantic similarity learning for image captioning evaluation, which uses contrastive learning to map entire sentences into a holistic semantic space where structural meaning is preserved.
Other researchers sought to break the single "black-box" score into interpretable components. Ming Jiang et al. introduced the REO framework in REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning. Instead of conflating all errors, REO explicitly disentangles caption quality into three distinct dimensions:
- Relevance: Is the generated content accurate to the scene?
- Extraness: Is the model hallucinating objects or details not present?
- Omission: Did the model miss critical focal points of the image?
This diagnostic view allows developers to understand exactly why a model is failing, rather than just knowing its n-gram overlap is low.
The Modern Era: Multi-Modal Verification and LLM Judges
With the rise of Large Language Models (LLMs) and Large Multimodal Models (LMMs), the paradigm has shifted from static string or graph matching to active, reference-free verification.
Hwanhee Lee et al. introduced an elegant information-theoretic approach called QACE in QACE: Asking Questions to Evaluate an Image Caption. The intuition behind QACE is that if a caption is factually accurate, it should serve as a reliable source of answers for questions about the image.
Candidate Caption: "A man is riding a bicycle down the street."
│
▼ (Question Generation)
Generated Question: "What is the man riding?"
│
▼ (Answer Verification)
Answer from Caption: "A bicycle"
Answer from Source Image: "A motorcycle"
│
▼ (Comparison)
MISMATH ──> Penalty for Factual Error
By generating questions from the candidate caption and verifying the answers against the source image or reference text, QACE turns evaluation into an explainable fact-checking pipeline. If the caption claims a man is riding a bicycle, but the image answers motorcycle, the metric identifies a factual hallucination that n-gram metrics would likely miss.
Today, as models generate increasingly long and detailed descriptions, evaluating captions as single semantic units is no longer viable. Zequn Zeng et al. address this in HICEScore: A Hierarchical Metric for Image Captioning Evaluation, arguing that detailed descriptions are collections of discrete claims about specific visual entities. Metrics like HICEScore and G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (developed by Tony Cheng Tong et al.) leverage advanced multimodal models and Chain-of-Thought reasoning to perform fine-grained, hierarchical verification. They decompose long captions into localized phrases, verifying the precision of individual claims against specific image regions.
By moving away from surface-level n-gram matching and embracing structural, semantic, and interactive verification, the field has created evaluation frameworks that truly measure whether a model understands the visual world it is trying to describe.
Key Foundational Work
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (Extended Abstract) (2015) — contributed 2 concept elements to this direction.
SPICE: Semantic Propositional Image Caption Evaluation (2016) — contributed 1 concept elements to this direction.
Semantic Tuples for Evaluation of Image to Sentence Generation (2015) — contributed 1 concept elements to this direction.