Quantifying the Amount of Visual Information Used by Neural Caption Generators

Marc Tanti, Albert Gatt, Kenneth P. Camilleri

Abstract

Evaluating neural image captioning models is traditionally a slow process, as standard metrics require the sequential generation of every word in a caption before comparison against human references. This paper introduces "pre-generation metrics," a novel approach that predicts model performance by analyzing the internal softmax probabilities a model assigns to reference captions. By bypassing the sequential generation step, evaluation time is reduced from over twenty minutes to under thirty seconds. We demonstrate that these metrics maintain a high correlation ($R^2 = 0.94$) with established post-generation metrics like CIDEr and METEOR across multiple datasets and architectures.

Introduction

In Image Description Generation (IDG), the standard evaluation pipeline relies on "post-generation" metrics such as BLEU, CIDEr, and METEOR. These metrics compare a fully generated sentence to a set of human-written reference captions. However, the sequential nature of neural text generation—requiring multiple forward passes per sentence—creates a significant computational bottleneck. For large-scale datasets like MSCOCO, this process can take upwards of 20 minutes on standard hardware, hindering rapid experimentation and real-time monitoring during training.

This research investigates whether a system's performance can be accurately assessed prior to the generation step by examining how the model "recognizes" gold-standard reference captions.

Methodology: The Pre-gen Framework

To derive a single quality score from internal model states, we propose a four-tier framework for "pre-gen metrics." Instead of generating new text, reference captions are fed into the model to extract the softmax probabilities for each constituent word. The framework then processes these probabilities through four stages:

  1. Filtering: Selecting which word probabilities to retain (e.g., only those where the model's top prediction matches the reference).
  2. Sentence Aggregation: Combining word-level data into a sentence score (e.g., calculating perplexity or prefix length).
  3. Image Aggregation: Merging scores from multiple reference captions for a single image (e.g., taking the mean or maximum).
  4. Dataset Aggregation: Consolidating image scores into a final dataset-wide metric.
The four-tier framework for calculating pre-generation metrics from raw word probabilities.

Through an exhaustive search of 504 unique metric combinations, we identified the "Off the Rails" hypothesis: the most effective predictor is the length of the longest prefix where the model’s most probable word matches the reference (prefix0). This suggests that early failures in the generation sequence are the primary drivers of poor final caption quality.

Experimental Setup

The framework was validated using three industry-standard benchmarks: Flickr8K, Flickr30k, and MSCOCO. To ensure the findings were not architecture-specific, the researchers tested 36 different neural captioning models, varying in their methods for merging visual and linguistic features.

Key Results and Efficiency

The "champion" metric identified, mean_max_normcount_prefix0, achieved a remarkable $R^2$ of 0.94 correlation with traditional post-generation metrics. This high correlation suggests that standard evaluation metrics essentially measure the same underlying sampling mechanisms that the neural network utilizes internally.

Correlation between Perplexity and standard post-generation metrics (CIDEr, SPICE, METEOR, WMD) across various datasets and architectures.

The primary advantage of this approach is efficiency. In our tests, computing a pre-gen metric like perplexity took only 28 seconds, compared to 20.8 minutes for full caption generation on the same hardware. Furthermore, the relationship between these internal probabilities and output quality remains broadly linear across different scales of model quality, making these metrics useful throughout the entire training lifecycle.

Conclusion and Future Work

Pre-generation metrics offer a robust and significantly faster alternative to traditional IDG evaluation. While they still require human-written reference captions, their predictive power and speed enable more efficient hyperparameter tuning. Future research will explore using these differentiable pre-gen metrics directly as loss functions to optimize models, potentially replacing complex reinforcement learning techniques like Self-Critical Sequence Training (SCST).

Summary updated

Abstract

This paper addresses the sensitivity of neural image caption generators to their visual input. A sensitivity analysis and omission analysis based on image foils is reported, showing that the extent to which image captioning architectures retain and are sensitive to visual information varies depending on the type of word being generated and the position in the caption as a whole. We motivate this work in the context of broader goals in the field to achieve more explainability in AI.
BibTeX
@inproceedings{tanti2018quantifying,
  author = {Marc Tanti and Albert Gatt and Kenneth P. Camilleri},
  title = {{Quantifying the Amount of Visual Information Used by Neural Caption Generators}},
  booktitle = {Workshops @ ECCV},
  year = {2018},
  url = {https://openreview.net/forum?id=B1Egi5WuWB},
}
Figure