Combine to Describe: Evaluating Compositional Generalization in Image Captioning

Georgios Pantazopoulos, Alessandro Suglia, Arash EshghiACL 2022

TL;DR

This paper investigates how well state-of-the-art image captioning models can generalize to novel combinations of concepts, extend their descriptions beyond typical lengths, and handle synonym substitutions. The authors find that current models struggle with systematic generalization and productivity, but show some robustness to synonym changes, highlighting a significant gap between human and AI language capabilities.

What Problem Does This Paper Address?

Compositionality, the ability to understand and generate complex ideas from simpler components, is a cornerstone of human language. While neural networks have achieved remarkable success in various Natural Language Processing (NLP) tasks, their ability to exhibit true compositional generalization remains a significant open question. Previous research has often focused on artificially constructed datasets or primarily on one aspect of compositionality (systematicity). This paper addresses this gap by proposing a comprehensive evaluation framework for image captioning models, a grounded NLP task, that examines compositional generalization across three key dimensions: systematicity, productivity, and substitutivity.

Methods

The authors adapt existing task-agnostic compositionality tests to the image captioning domain. They create custom training and evaluation splits from the MSCOCO dataset for each test.

  • Systematicity: Evaluates the model's ability to combine concepts that were observed independently during training but not together. For example, if a model has seen "black dogs" and "white cats," can it describe a "white dog"?
  • Productivity: Assesses the model's capacity to generate captions that extend beyond the length distribution observed during training, and to describe images with higher visual density (more objects).
  • Substitutivity: Tests the model's robustness to synonym substitutions, checking if replacing a word with its synonym alters the meaning or structure of the generated caption.

The evaluation uses standard captioning metrics (BLEU, METEOR, ROUGE, CIDEr) and semantic similarity metrics (BERTSCORE). They also analyze the impact of different training strategies, specifically comparing standard cross-entropy (XE) optimization with Self-Critical Sequence Training (SCST).

Datasets

  • MSCOCO dataset: Used for both training and evaluation. Custom splits were created for each of the three compositionality tests.
  • No new benchmarks were introduced, but existing concepts and categories from prior work (e.g., Lu et al., 2018 for COCO categories) were utilized.

Key Insights and Ideas

  • Compositionality is multifaceted: Evaluating only one aspect (like systematicity) can be misleading. A comprehensive approach is needed.
  • Current models lack true systematic generalization: Despite advances, image captioning models struggle to combine concepts in novel ways that were not explicitly seen during training.
  • SCST hinders systematicity: Surprisingly, fine-tuning with SCST, which is often used to improve captioning metrics, appears to worsen systematic generalization compared to standard XE training. This is attributed to the reward function potentially penalizing novel but plausible descriptions.
  • Productivity is a challenge: Models have difficulty generating captions that are significantly longer than those seen during training, and they also struggle with describing images with higher visual density.
  • Models are somewhat robust to synonyms: While performance drops are observed, models show a degree of robustness when fine-grained object descriptions are replaced with more general synonyms.
  • Word overlap metrics can be misleading: Metrics like BLEU and CIDEr might favor models that produce sequences closer to the target, potentially at the expense of true compositional understanding or groundedness.

Key Results

  • Systematicity: On average, only about 4.57% (XE) to 4.65% (SCST) of the time did the models include the target concept pair in their top caption. SCST fine-tuning showed a slight decrease in performance for systematicity compared to XE.
  • Productivity: Models showed a significant performance drop on metrics when evaluated on images with longer ground truth captions (Test Rich) compared to images with lengths similar to the training data (Test Base). A similar drop was observed for visual density, though the semantic degradation was less pronounced than with caption length.
  • Substitutivity: While there were performance drops after synonym substitution, the models generally maintained semantic equivalence, suggesting they can handle some level of synonym replacement. SCST generally showed better performance than XE.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current models struggle with systematic generalization and productivity. The SCST training, while improving some metrics, negatively impacts systematicity. The substitutivity test primarily focused on replacing fine-grained terms with more abstract ones, not the reverse.
  • Future Work: The authors suggest modifying reward functions to avoid penalizing plausible descriptions that deviate from the ground truth. They also propose exploring the substitution of generic descriptions with more fine-grained categories. Furthermore, they advocate for multimodal metrics that consider both text and vision, and for training regimes that encourage models to explore the search space and generate sequences beyond average lengths.

Summary updated

BibTeX
@inproceedings{pantazopoulos2022combine,
  author = {Georgios Pantazopoulos and Alessandro Suglia and Arash Eshghi},
  title = {{Combine to Describe: Evaluating Compositional Generalization in Image Captioning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure