Abstract
This paper introduces Annotated 3D Shapes (A3DS), a comprehensive dataset designed to evaluate the "pragmatic" capabilities of image captioning models—their ability to provide informative, relevant, and efficient descriptions in context. While standard models are often "task-neutral" and overly descriptive, this study demonstrates that fine-tuning in a multi-agent communicative environment allows models to develop human-like efficiency and descriptive biases. Our results show a significant increase in contrastive efficiency, though at a slight cost to raw discriminative accuracy.
Introduction
Human communication is inherently context-dependent, governed by "Gricean maxims" that encourage speakers to be truthful, informative, relevant, and brief. However, most AI image captioning models are trained to be "truthful" without regard for utility. For instance, describing a red ball as "a red ball" is sufficient in most cases, but if it sits next to a blue ball, "the red one" is more efficient. Evaluating these pragmatic abilities has been hindered by a lack of datasets with exhaustive feature labels. To address this, we present the A3DS benchmark, where every image feature is known and labeled, enabling precise mathematical evaluation of communicative efficiency.
The A3DS Benchmark and Methodology
The Annotated 3D Shapes (A3DS) dataset was constructed from 480,000 images of geometric objects. Using a structured grammar, the researchers generated over 9 million exhaustive captions (listing all properties) and 12 million short captions. This synthetic approach provides a "gold standard" for testing model logic.
We evaluated a standard pretrained baseline (B) against two models fine-tuned via Reinforcement Learning (RL) in a "sender-listener" game: Random Pairs (RP) and Similar Pairs (SP). In this setup, the "Sender" (captioner) must describe a target image so a "Listener" can distinguish it from a distractor.
To quantify performance, four key metrics were derived:
- Discriminativity ($d$): The ability of the caption to uniquely identify the target.
- Contrastive Efficiency ($e$): The avoidance of over-description.
- Relevance ($r$): The exclusion of features that do not aid in distinguishing the target.
- Optimal Discriminativity ($od$): A binary measure of whether a caption is both accurate and efficient.
Results and Analysis
The experiments reveal a stark "Truth vs. Utility" trade-off. While the baseline model is highly truthful, it is a poor communicator, often providing a "wall of text" that lacks pragmatic focus. In contrast, fine-tuned models (RP and SP) demonstrated a massive leap in Contrastive Efficiency, rising from approximately 0.20 to 0.88.
As shown in Table 1, the gain in efficiency comes with a slight reduction in raw Discriminativity (dropping from ~0.99 to ~0.82). In their pursuit of brevity, the models occasionally became too vague, leading to identification errors by the listener.
Emergent Descriptive Biases
A key finding is the emergence of human-like biases during fine-tuning. Models developed a distinct shape bias, preferring to mention an object's shape even when other features like color or scale would suffice for discrimination. This mirrors human cognitive patterns where certain features are more "salient."
As illustrated in Figure 1, the models consistently over-reported shape and object color across one, two, and three-feature conditions. Conversely, they struggled with less salient features like orientation or floor color.
Conclusion and Future Work
While fine-tuning in communicative environments significantly improves pragmatic utility, it also introduces challenges such as "language drift," where models develop efficient but non-human-readable codes. Furthermore, the synthetic nature of A3DS limits its immediate applicability to complex real-world scenes. Future research will focus on expanding these metrics to Large Language Models (LLMs) and more complex visual environments to further bridge the gap between AI truthfulness and communicative effectiveness.
Summary updated