Abstract
Evaluating image captioning models remains a significant challenge, as traditional n-gram metrics like BLEU or CIDEr often fail to capture semantic nuances. While data-driven metrics like CLIPScore offer improvements, they frequently struggle with "hallucinations"—describing objects not present in the image—and suffer from a domain gap between pre-training tasks and human evaluation criteria. We introduce Polos, a supervised evaluation metric trained using a novel Multimodal Metric Learning from Human Feedback ($M^2LHF$) framework. Leveraging the new Polaris dataset, which contains 131,020 human judgments, Polos significantly improves alignment with human perception and excels at detecting captioning errors.
Introduction
The primary hurdle in image captioning research is the lack of a robust, automated evaluator that mirrors human judgment. Existing metrics often reward captions that share words with a reference but fail to describe the image accurately, or they rely on embeddings that are insensitive to fine-grained visual-textual discrepancies. Furthermore, the scarcity of large-scale human-annotated datasets for metric training has limited the development of generalizable supervised evaluators.
Polos addresses these issues by training on a dataset ten times larger than previous standards and utilizing a multimodal fusion architecture designed to highlight divergences between candidate captions, reference text, and the source image.
The $M^2LHF$ Framework and Polos
The core of Polos is the Multimodal Metric Learning from Human Feedback ($M^2LHF$) framework. This architecture enables the model to compute a quality score by integrating features from three sources: the image, the candidate caption, and one or more reference captions.
Technical Components
- Dual Text Encoders: Polos utilizes CLIP for initial vision-language alignment and RoBERTa (enhanced with SimCSE) for deeper linguistic analysis. This combination ensures the model captures both visual grounding and complex semantic relationships within the text.
- Parallel Feature Extraction: Rather than simple feature concatenation, Polos computes the difference and the Hadamard product between the candidate caption and the references/images. This mechanism effectively "spotlights" inconsistencies, making the model highly sensitive to hallucinations.
- Regression-Based Scoring: Polos uses a Multi-Layer Perceptron (MLP) to regress directly to a continuous human-like score (0 to 1). This approach better handles the subjectivity inherent in having multiple valid reference descriptions compared to simple ranking models.
The Polaris Dataset
To train and validate Polos, the authors curated Polaris, the largest dataset of its kind. It features:
- 131,020 human judgments provided by 550 evaluators.
- Captions generated by 10 state-of-the-art models (including BLIP-2 and GIT) across diverse image sets like MS-COCO and nocaps.
- Comprehensive benchmarking against established datasets such as Composite, Flickr8K, and FOIL.
Key Results and Insights
Polos sets a new standard for caption evaluation across several dimensions:
- Human Alignment: Polos achieved the highest correlation with human judgments on nearly all tested benchmarks, outperforming the next best metric by 1.8 points on the Polaris dataset.
- Hallucination Detection: On the FOIL benchmark, which tests the ability to detect single-word factual errors, Polos reached an accuracy of 95.4%, significantly surpassing both traditional and CLIP-based metrics.
- Computational Efficiency: Despite its multimodal architecture, Polos is 2.4x faster than the traditional SPICE metric, facilitating its use in large-scale model development pipelines.
A key insight from this work is that text-only or image-only metrics are insufficient for high-fidelity evaluation. The success of Polos stems from its ability to model the intricate, simultaneous relationships between text-image pairs and candidate-reference pairs.
Limitations and Future Directions
While Polos represents a significant leap forward, it occasionally overestimates the quality of "safe," generic captions (e.g., "a dog") that lack specific details present in the image. Additionally, relying on global CLIP embeddings can sometimes lead to missing very fine-grained errors in highly complex scenes. Future work will explore incorporating region-based features, such as those from RegionCLIP, to further refine the alignment between specific image regions and descriptive text spans.
Summary updated