TL;DR
This paper introduces VELA, a novel LLM-hybrid evaluation metric designed to accurately assess the quality of long, detailed image captions. VELA addresses the limitations of existing metrics that struggle with lengthy descriptions and proposes a new benchmark, LongCap-Arena, to facilitate research in this area.
What Problem Does This Paper Address?
The field of multimodal large language models (MLLMs) has seen rapid advancements, leading to impressive capabilities in generating long and detailed image captions. However, evaluating these captions presents a significant challenge. Traditional metrics like BLEU and CIDEr, primarily designed for shorter captions, show weak correlation with human judgments when applied to longer ones. More recent "LLM-as-a-Judge" approaches, while promising, suffer from prohibitively slow inference times due to their autoregressive nature and early fusion of visual information. This inefficiency hinders their practical use in developing and refining MLLMs. The paper aims to bridge this gap by proposing an evaluation metric that is both computationally efficient and highly aligned with human perception of caption quality for long descriptions.
Methods
VELA employs a novel "LLM-Hybrid-as-a-Judge" framework, combining the strengths of large language models with efficient computational techniques. It consists of two main branches:
- R2C-LLM Branch: This branch uses a lightweight, non-autoregressive LLM (Qwen2.5-3B) to capture the linguistic relationships between a candidate caption and human-provided reference captions. Crucially, it utilizes a "late fusion" approach, integrating visual information only after processing text, which helps maintain computational efficiency and avoids increasing input sequence lengths.
- I2C-Align Branch: This branch leverages Long-CLIP, a modified version of CLIP capable of processing longer text inputs, to directly assess the similarity between the image and the candidate caption. This branch operates independently of MLLMs, further contributing to speed.
The outputs from these two branches are then combined and passed through a learned projection layer to produce scores for three distinct perspectives: Descriptiveness, Relevance, and Fluency. This multi-perspective scoring is a key advantage over single-score metrics.
Datasets
The authors introduce LongCap-Arena, a new benchmark specifically designed for evaluating long image captions. It comprises:
- 7,805 images.
- Corresponding human-provided long reference captions.
- Long candidate captions generated by ten representative MLLMs and image captioning models.
- 32,246 human judgments collected from 1,020 annotators, assessing captions from three perspectives: Descriptiveness, Relevance, and Fluency.
The benchmark is built upon the DCI dataset, ensuring high-quality reference captions. The training, validation, and test splits of VELA are derived from the DCI dataset to prevent data leakage.
Key Insights and Ideas
- Late Fusion for Efficiency: By adopting a late fusion strategy for visual information, VELA significantly reduces inference time compared to MLLM-based methods that use early fusion. This is a critical step towards practical evaluation.
- Multi-Perspective Evaluation: Scoring captions across Descriptiveness, Relevance, and Fluency provides a more nuanced and comprehensive assessment than single-score metrics, aligning better with human judgment.
- Long-Context Vision-Language Models: The development of LongCap-Arena highlights the growing need for specialized benchmarks to evaluate the increasingly sophisticated long-caption generation capabilities of MLLMs.
- "Superhuman" Performance: VELA not only outperforms existing metrics but also achieves performance surpassing human annotators on certain aspects (Desc. and Flu.) on the LongCap-Arena benchmark, suggesting its potential as a reliable automated evaluator.
Key Results
VELA demonstrated strong performance on the LongCap-Arena benchmark, achieving scores of 56.4 (Desc.), 40.0 (Rel.), and 57.4 (Flu.) on TestA, and 54.0 (Desc.), 52.3 (Rel.), and 39.0 (Flu.) on TestB. These scores were significantly higher than baseline metrics and even surpassed human performance in Descriptiveness and Fluency.
Crucially, VELA achieved an inference time of approximately 258 ms per sample, which is about five times faster than existing LLM-based metrics that can take over 1000 ms.
Qualitative analysis showed VELA's ability to provide more accurate scores than GPT-40 in cases where GPT-40 overestimated caption quality. Ablation studies confirmed the importance of both the R2C-LLM and I2C-Align branches, with the Long-CLIP backbone being particularly crucial for the I2C-Align branch.
Limitations and Future Work
The paper acknowledges several limitations:
- Reference Quality: The metric's performance can be affected by the quality and detail of the reference captions.
- Object Salience: VELA may sometimes overlook semantically important objects, especially if they occupy small regions in the image.
- R2C-LLM Access: The reliance on accessing last hidden states in the R2C-LLM branch prevents the direct use of closed-source LLMs.
Future work could involve developing mechanisms to better integrate visual information with language models, similar to gated attention layers in models like Flamingo, to improve performance, particularly in Relevance. The authors also suggest exploring improvements to the I2C-Align branch for better grounding.
Summary updated