Abstract
Evaluating neural image captioning models is traditionally a slow process, as standard metrics require the sequential generation of every word in a caption before comparison against human references. This paper introduces "pre-generation metrics," a novel approach that predicts model performance by analyzing the internal softmax probabilities a model assigns to reference captions. By bypassing the sequential generation step, evaluation time is reduced from over twenty minutes to under thirty seconds. We demonstrate that these metrics maintain a high correlation ($R^2 = 0.94$) with established post-generation metrics like CIDEr and METEOR across multiple datasets and architectures.
Introduction
In Image Description Generation (IDG), the standard evaluation pipeline relies on "post-generation" metrics such as BLEU, CIDEr, and METEOR. These metrics compare a fully generated sentence to a set of human-written reference captions. However, the sequential nature of neural text generation—requiring multiple forward passes per sentence—creates a significant computational bottleneck. For large-scale datasets like MSCOCO, this process can take upwards of 20 minutes on standard hardware, hindering rapid experimentation and real-time monitoring during training.
This research investigates whether a system's performance can be accurately assessed prior to the generation step by examining how the model "recognizes" gold-standard reference captions.
Methodology: The Pre-gen Framework
To derive a single quality score from internal model states, we propose a four-tier framework for "pre-gen metrics." Instead of generating new text, reference captions are fed into the model to extract the softmax probabilities for each constituent word. The framework then processes these probabilities through four stages:
- Filtering: Selecting which word probabilities to retain (e.g., only those where the model's top prediction matches the reference).
- Sentence Aggregation: Combining word-level data into a sentence score (e.g., calculating perplexity or prefix length).
- Image Aggregation: Merging scores from multiple reference captions for a single image (e.g., taking the mean or maximum).
- Dataset Aggregation: Consolidating image scores into a final dataset-wide metric.
Through an exhaustive search of 504 unique metric combinations, we identified the "Off the Rails" hypothesis: the most effective predictor is the length of the longest prefix where the model’s most probable word matches the reference (prefix0). This suggests that early failures in the generation sequence are the primary drivers of poor final caption quality.
Experimental Setup
The framework was validated using three industry-standard benchmarks: Flickr8K, Flickr30k, and MSCOCO. To ensure the findings were not architecture-specific, the researchers tested 36 different neural captioning models, varying in their methods for merging visual and linguistic features.
Key Results and Efficiency
The "champion" metric identified, mean_max_normcount_prefix0, achieved a remarkable $R^2$ of 0.94 correlation with traditional post-generation metrics. This high correlation suggests that standard evaluation metrics essentially measure the same underlying sampling mechanisms that the neural network utilizes internally.
The primary advantage of this approach is efficiency. In our tests, computing a pre-gen metric like perplexity took only 28 seconds, compared to 20.8 minutes for full caption generation on the same hardware. Furthermore, the relationship between these internal probabilities and output quality remains broadly linear across different scales of model quality, making these metrics useful throughout the entire training lifecycle.
Conclusion and Future Work
Pre-generation metrics offer a robust and significantly faster alternative to traditional IDG evaluation. While they still require human-written reference captions, their predictive power and speed enable more efficient hyperparameter tuning. Future research will explore using these differentiable pre-gen metrics directly as loss functions to optimize models, potentially replacing complex reinforcement learning techniques like Self-Critical Sequence Training (SCST).
Summary updated