RONA: Pragmatically Diverse Image Captioning with Coherence Relations

Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon LeeNAACL 2025

TL;DR

Current AI image captioning models often produce diverse captions by tweaking vocabulary or focusing on different image parts. However, they miss the nuanced, pragmatic ways humans convey meaning. This paper introduces RONA, a novel prompting strategy for multimodal large language models (MLLMs) that leverages "Coherence Relations" to generate more diverse and contextually relevant captions, bridging the gap between literal descriptions and human-like communication.

What Problem Does This Paper Address?

Writing assistants powered by AI, like Grammarly or Microsoft Copilot, are becoming increasingly sophisticated, helping users with a wide range of writing tasks. Image captioning, a key task for these assistants, has seen significant progress with the advent of multimodal large language models (MLLMs). While these models can accurately describe visual content, real-world captions often go beyond mere description. They convey a central message, provide context, and offer different perspectives, leading to a richer and more diverse output. Existing methods for increasing caption diversity primarily focus on syntactic variations (different words) or semantic variations (highlighting different image elements). However, these approaches often fail to capture the subtle, pragmatic nuances that humans employ, such as implicatures and metaphors, to make captions more engaging and relatable. This paper aims to address this limitation by enabling MLLMs to generate pragmatically diverse captions that better align with human communication styles.

Methods

The core innovation of RONA lies in its use of Coherence Relations (CRs) as a controllable axis for generating pragmatic variations in image captions. Inspired by Discourse Theory, CRs provide a structured way to model the relationships between an image and its caption, encompassing both contextual and pragmatic aspects. RONA utilizes five specific types of CRs:

  • Insertion: The caption describes a salient object that isn't explicitly mentioned in the image.
  • Concretization: Both the image and caption mention a key visual entity, with the caption adding more meaning about its context.
  • Projection: The main entity in the caption is implicitly related to the visual elements in the image.
  • Restatement: The caption directly describes the image's overall context, offering a more detailed view.
  • Extension: The caption elaborates on the visual scene, introducing new ideas or stories to fill narrative gaps.

RONA employs an in-context learning approach, where simplified definitions of these CRs are provided as system prompts to the MLLM. For each image, RONA generates one caption per CR, resulting in a set of five pragmatically diverse captions.

Datasets

The authors evaluated RONA on two datasets representing real-world scenarios for writing assistants:

  • Tweet Subtitles: This dataset contains 16,000 image-text pairs sourced from Twitter, cleaned to remove low-quality samples. The evaluation used the entire test set of 1,600 samples.
  • ANNA (Abstractive News Captions): This dataset comprises 29,625 image-text pairs from The New York Times, focusing on non-Named Entity objects. For evaluation, a random sample of 1,500 images from the test set was used.

Both datasets feature "abstractive" or non-descriptive captions, making them suitable for evaluating pragmatic diversity.

Key Insights and Ideas

  • Coherence Relations as a Pragmatic Axis: The paper demonstrates that CRs offer a structured and effective way to control pragmatic variations in caption generation, going beyond simple syntactic or semantic changes.
  • Bridging the Gap to Human Communication: RONA's success suggests that MLLMs can be guided to produce captions that are not just descriptive but also convey intent and perspective, mimicking human communication more closely.
  • Image-Only vs. Image+Caption Tasks: The study highlights that while image-only captioning relies heavily on visual features, the Image+Caption task reveals the MLLMs' ability to leverage both modalities for more expressive and diverse outputs, underscoring the need for such evaluations.
  • Diversity Without Sacrificing Relevance: A key finding is that RONA generates more diverse captions without compromising their similarity to the ground truth or their contextual relevance, as measured by metrics like BLEURT and CLIPScore.

Key Results

The evaluation involved two popular MLLMs, GPT-4o and Claude 3.5 Sonnet v2, across two tasks (Image-only and Image+Caption) and two datasets.

  • Improved Diversity: RONA-based models outperformed their respective baselines in 7 out of 8 settings for both diversity and ground-truth similarity. Specifically, RONA-based models were rated as more diverse by the Self-BLEURT and Div-2 metrics in over half of the tested scenarios.
  • Maintained Relevance: Metrics for image-caption similarity (BLEURT and CLIPScore) showed that RONA-based models were preferred in 7 out of 8 and 6 out of 8 settings, respectively, indicating that increased diversity did not come at the cost of contextual relevance.
  • Qualitative Examples: The appendix provides illustrative examples showing how RONA generates captions with distinct pragmatic flavors for the same image, such as "The perfect way to end a summer evening on the patio" (Insertion) versus "Vanilla ice cream garnished with crispy honeycomb pieces in a blue ceramic bowl" (Concretization) for an ice cream dessert.

Limitations and Future Work

  • Limited MLLM Architectures: The current study focused on a few top-performing proprietary MLLMs. Future work could explore how open-source MLLMs leverage CRs for image captioning.
  • Prompt Following Accuracy: The paper acknowledges that validating the MLLMs' adherence to specific CRs is a challenge, especially for smaller models. Inaccuracies could lead to hallucinations and factual inconsistencies.
  • Bias and Stereotypes: The use of CRs, which leverage common-sense knowledge, could potentially surface model biases and stereotypes, particularly with culturally sensitive content.
  • Future Directions: The authors suggest incorporating factual consistency metrics and human preference ratings to identify and mitigate hallucinations. They also advocate for responsible use of writing assistants with human oversight.

Summary updated

BibTeX
@inproceedings{ramakrishnan2025rona,
  author = {Aashish Anantha Ramakrishnan and Aadarsh Anantha Ramakrishnan and Dongwon Lee},
  title = {{RONA: Pragmatically Diverse Image Captioning with Coherence Relations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure