Evaluating pragmatic abilities of image captioners on A3DS

Polina Tsvilodub, Michael Franke

Abstract

This paper introduces Annotated 3D Shapes (A3DS), a comprehensive dataset designed to evaluate the "pragmatic" capabilities of image captioning models—their ability to provide informative, relevant, and efficient descriptions in context. While standard models are often "task-neutral" and overly descriptive, this study demonstrates that fine-tuning in a multi-agent communicative environment allows models to develop human-like efficiency and descriptive biases. Our results show a significant increase in contrastive efficiency, though at a slight cost to raw discriminative accuracy.

Introduction

Human communication is inherently context-dependent, governed by "Gricean maxims" that encourage speakers to be truthful, informative, relevant, and brief. However, most AI image captioning models are trained to be "truthful" without regard for utility. For instance, describing a red ball as "a red ball" is sufficient in most cases, but if it sits next to a blue ball, "the red one" is more efficient. Evaluating these pragmatic abilities has been hindered by a lack of datasets with exhaustive feature labels. To address this, we present the A3DS benchmark, where every image feature is known and labeled, enabling precise mathematical evaluation of communicative efficiency.

The A3DS Benchmark and Methodology

The Annotated 3D Shapes (A3DS) dataset was constructed from 480,000 images of geometric objects. Using a structured grammar, the researchers generated over 9 million exhaustive captions (listing all properties) and 12 million short captions. This synthetic approach provides a "gold standard" for testing model logic.

We evaluated a standard pretrained baseline (B) against two models fine-tuned via Reinforcement Learning (RL) in a "sender-listener" game: Random Pairs (RP) and Similar Pairs (SP). In this setup, the "Sender" (captioner) must describe a target image so a "Listener" can distinguish it from a distractor.

To quantify performance, four key metrics were derived:

  • Discriminativity ($d$): The ability of the caption to uniquely identify the target.
  • Contrastive Efficiency ($e$): The avoidance of over-description.
  • Relevance ($r$): The exclusion of features that do not aid in distinguishing the target.
  • Optimal Discriminativity ($od$): A binary measure of whether a caption is both accurate and efficient.

Results and Analysis

The experiments reveal a stark "Truth vs. Utility" trade-off. While the baseline model is highly truthful, it is a poor communicator, often providing a "wall of text" that lacks pragmatic focus. In contrast, fine-tuned models (RP and SP) demonstrated a massive leap in Contrastive Efficiency, rising from approximately 0.20 to 0.88.

Table 1: Performance metrics comparing the baseline (B) to fine-tuned models (RP, SP) across different feature conditions.

As shown in Table 1, the gain in efficiency comes with a slight reduction in raw Discriminativity (dropping from ~0.99 to ~0.82). In their pursuit of brevity, the models occasionally became too vague, leading to identification errors by the listener.

Emergent Descriptive Biases

A key finding is the emergence of human-like biases during fine-tuning. Models developed a distinct shape bias, preferring to mention an object's shape even when other features like color or scale would suffice for discrimination. This mirrors human cognitive patterns where certain features are more "salient."

Figure 1: Proportion of redundant mentions by feature, illustrating the emergent preference for shape and object color in communicative contexts.

As illustrated in Figure 1, the models consistently over-reported shape and object color across one, two, and three-feature conditions. Conversely, they struggled with less salient features like orientation or floor color.

Conclusion and Future Work

While fine-tuning in communicative environments significantly improves pragmatic utility, it also introduces challenges such as "language drift," where models develop efficient but non-human-readable codes. Furthermore, the synthetic nature of A3DS limits its immediate applicability to complex real-world scenes. Future research will focus on expanding these metrics to Large Language Models (LLMs) and more complex visual environments to further bridge the gap between AI truthfulness and communicative effectiveness.

Summary updated

Abstract

Evaluating grounded neural language model performance with respect to pragmatic qualities like the trade off between truthfulness, contrastivity and overinformativity of generated utterances remains a challenge in absence of data collected from humans. To enable such evaluation, we present a novel open source image-text dataset "Annotated 3D Shapes" (A3DS) comprising over nine million exhaustive natural language annotations and over 12 million variable-granularity captions for the 480,000 images provided by Burges & Kim (2018). We showcase the evaluation of pragmatic abilities developed by a task-neutral image captioner fine-tuned in a multi-agent communication setting to produce contrastive captions. The evaluation is enabled by the dataset because the exhaustive annotations allow to quantify the presence of contrastive features in the model's generations. We show that the model develops human-like patterns (informativity, brevity, over-informativity for specific features (e.g., shape, color biases)).
BibTeX
@inproceedings{tsvilodub2023evaluating,
  author = {Polina Tsvilodub and Michael Franke},
  title = {{Evaluating pragmatic abilities of image captioners on A3DS}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.110},
}
Figure