“Caption” as a Coherence Relation: Evidence and Implications

Malihe Alikhani, Matthew StoneNAACL 2019

TL;DR

This paper argues that current image captioning datasets are systematically limited in their verb usage, primarily focusing on atelic, present-tense descriptions of visible events. This limitation stems from the specific "caption" coherence relation, which constrains how text describes images, and hinders the development of more general vision-language models.

What Problem Does This Paper Address?

Researchers are increasingly using multimodal corpora (combining text and images) to build models that understand the relationship between language and the visual world. A key task is generating descriptive captions for images. However, this paper identifies a significant problem: existing image-caption datasets, like MSCOCO and Flickr30K, exhibit a narrow range of verb usage. This uniformity, the authors argue, is not representative of how language can describe visual content more broadly and limits the applicability of models trained on these datasets to tasks beyond simple description. They propose that this limitation is tied to the specific "caption" coherence relation, which imposes certain constraints on the descriptions.

Methods

The authors conduct an empirical study analyzing verb usage across several prominent image-text corpora: COCO, Flickr30K, VIST, Conceptual Captions (CC), and a Recipe dataset. They compare these to general English corpora like the American National Corpus (ANC) and Facebook's children's stories (FS). Their analysis involves:

  • Part-of-Speech Tagging and Parsing: Using the SpaCy toolkit to identify verbs and their auxiliaries.
  • Verb Classification: Categorizing verbs based on tense (past/non-past), aspect (simple, progressive, perfect), and lexical aspect (stative, atelic, telic).
  • Frequency Analysis: Quantifying the prevalence of different verb types and lemmas across corpora.
  • Annotation Study: Manually annotating a sample of captions to determine the proportion of stative, atelic, and telic descriptions.

Datasets

The paper analyzes the following datasets:

  • Microsoft Common Objects in Context (COCO): A widely used dataset for image captioning.
  • Flickr30K: Another popular dataset for image captioning.
  • Visual Storytelling (VIST): A dataset designed for generating short narratives from sequences of images.
  • Google's Conceptual Captions (CC): A large dataset of image-text pairs derived from web images and their alt-text.
  • Recipe Dataset: A dataset of multimodal cooking recipes.

These are compared against:

  • American National Corpus (ANC): A balanced corpus of spoken and written English.
  • Facebook's children's stories (FS): A corpus of written narrative.
  • Google Ngram Dataset: Used for comparing verb frequencies in broader text.

Key Insights and Ideas

  • "Caption" as a Coherence Relation: The authors propose that "captioning" is a specific coherence relation between text and image, distinct from others like illustration or narrative. This relation imposes specific constraints on the text.
  • Systematic Verb Deficits in Captions: Image-caption corpora are characterized by a disproportionately limited and less diverse verb distribution compared to general text corpora.
  • Preference for Atelic, Present-Tense, Progressive Verbs: Captions predominantly use verbs that describe ongoing, unbounded activities (atelic) in the present tense and progressive aspect, as if the event is happening "now" and has been happening indefinitely.
  • Visible Events are Key: Captions tend to focus on visually observable events, excluding verbs related to internal states, beliefs, or abstract concepts.
  • Implications for Model Training: The uniformity of verb usage in caption datasets means models trained on them will struggle with describing extended activities or events with clear endpoints, limiting their generalizability.
  • Broader Corpora Offer More Diversity: Datasets like the Recipe dataset, which involve more complex relationships between text and images (e.g., instructions, narratives), exhibit a richer verb usage.

Key Results

  • Verb Frequency Drop: Verbs constitute a much smaller percentage of tokens in image-caption corpora (e.g., 0.012 in Flickr) compared to ANC (0.184).
  • Dominance of Frequent Verbs: A small set of "Frequent Caption Verbs" (FCVs) like "is/are," "wearing," "sitting," "standing," "has/have," "walking," "holding," "looking," "playing," "smiling," "jumping," "watching," "talking," "doing," "eating," and "carrying" dominate caption corpora, making up a significant portion of verb usage.
  • Atelic Preference: An annotation study found that caption corpora contain significantly fewer telic verbs (describing events with an endpoint) compared to general corpora like ANC and Recipe. For example, Flickr and COCO had a telic rate of around 22-23% out of 500 samples, while ANC had 197 telic verbs.
  • Progressive Aspect Dominance: Progressive verbs make up nearly 49% of verbs in COCO and Flickr, a stark contrast to ANC's ~8%. Models trained on COCO also show a strong preference for progressive forms.

Limitations and Future Work

  • Limitations of Caption Corpora: The primary limitation identified is the narrow scope of verb usage in current image-caption datasets, which restricts the ability of models to describe a wider range of visual events. This is attributed to the inherent constraints of the "caption" coherence relation.
  • Future Work: The authors recommend that future image-text corpora should:
  • Increase Diversity: Explicitly aim to increase the diversity of event descriptions.
  • Explore Diverse Relations: Investigate and model a broader range of coherence relations between text and imagery beyond simple captioning.
  • Improve Theoretical Resources: Develop better theoretical and computational resources for verb classification to capture perceptual and temporal inference more accurately.
  • Broader Data Collection: Train models on more diverse datasets, including those with illustrations and more complex narrative structures, to improve generalization.

Summary updated

BibTeX
@inproceedings{alikhani2019caption,
  author = {Malihe Alikhani and Matthew Stone},
  title = {{“Caption” as a Coherence Relation: Evidence and Implications}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure