TL;DR
This paper investigates how human speakers adjust their language when referring to objects in a visual dialogue. They found that utterances become less descriptive over time but remain equally effective at identifying the target object, suggesting a reliance on dialogue history and a focus on the most salient information. This work leverages CLIP to quantify these linguistic properties, offering a new way to analyze human communication strategies.
What Problem Does This Paper Address?
When we talk to each other, especially when pointing out specific things in a shared visual scene, our language changes. Initially, we might be very descriptive, but as the conversation progresses and we establish common ground, our descriptions tend to get shorter. The big question is: how do we manage to still be understood even when we're being less descriptive? Are we just relying on the fact that the other person knows what we're talking about from previous turns, or are we getting better at picking out just the essential details that make an object unique in that specific moment? This paper aims to quantify these two aspects – descriptiveness (how well an utterance describes an image in isolation) and discriminativeness (how well it helps pick out the correct image from a set of similar ones) – in human referring utterances within dialogues, and to understand how they evolve.
Methods
The core of this research is the use of CLIP (Contrastive Language-Image Pre-training), a powerful multimodal model. CLIP is trained to align images and text, making it adept at understanding the relationship between visual content and linguistic descriptions. The authors use CLIP in a "zero-shot" manner, meaning they don't fine-tune it on their specific dialogue data, but rather leverage its pre-trained knowledge.
To measure descriptiveness, they feed individual referring utterances and their corresponding target images into CLIP. The model then outputs a score (CLIPScore) representing how well the text describes the image, treating each utterance in isolation.
For discriminativeness, they simulate a reference resolution task. CLIP is given a referring utterance along with a set of distractor images (the visual context). The model's ability to assign the highest probability to the correct target image is measured as accuracy. They also analyze the entropy of CLIP's predictions, which indicates how "uncertain" the model is about which image is the correct referent.
Datasets
The primary dataset used is PhotoBook (PB), a multimodal task-oriented dialogue dataset. PB involves players trying to find common images without seeing each other's visual contexts. The authors focus on a subset called PB-GOLD, which contains manually annotated referring utterances and referential chains. This subset, while smaller, is considered higher quality.
For comparison, they also use:
- COCO captions: Multiple captions for images, where the order of captions is arbitrary.
- Image Description Sequences (IDS): Descriptions of images that are incrementally built up, adding more details over time.
Key Insights and Ideas
- The "Less is More" Paradox: Speakers become less descriptive over time, but their utterances remain just as effective at distinguishing the target object from others. This suggests a trade-off where conciseness is achieved without sacrificing clarity.
- Dialogue History as a Crutch: When utterances become less descriptive, relying on the previous turn in the dialogue significantly reduces the model's (and presumably the human speaker's) uncertainty in identifying the referent. This highlights the importance of shared context in communication.
- Distilling Discriminative Information: Speakers seem to actively select words that are most discriminative for the target image, even if those words are less descriptive overall. This implies a strategic selection of language to maximize communicative efficiency.
- CLIP as a Cognitive Probe: The study demonstrates CLIP's utility not just for generating text or images, but also for analyzing human communication patterns and potentially understanding cognitive processes in language use.
Key Results
- Descriptiveness Decreases: CLIPScore for referring utterances shows a statistically significant decrease across dialogue ranks (utterance positions in a chain).
- Discriminativeness Remains Stable: Reference resolution accuracy using CLIP remains high (around 80%) and does not significantly change across ranks, even when utterances become less descriptive.
- Entropy Increases with Less Description: As utterances become less descriptive, the entropy of CLIP's predictions increases, indicating higher uncertainty when the utterance is considered in isolation.
- Dialogue History Reduces Uncertainty: Including the previous utterance in the input to CLIP significantly reduces entropy, demonstrating its role in maintaining communicative success.
- Discriminative Words are Key: For a remarkable 60% of utterances, at least one of the top-10 most discriminative words identified by CLIP is present in the utterance.
Limitations and Future Work
- Model vs. Human: While CLIP is used to analyze human strategies, it's important to remember that CLIP is a model, not a perfect replica of human cognition. The authors acknowledge that their approach offers insights into "human-like outcomes" rather than a direct simulation.
- Future Exploration: The authors suggest exploring novel ways to integrate CLIP or its representations into more sophisticated models that explicitly embed dialogue history and visual context for reference resolution or generation. This could lead to more human-like conversational agents.
Summary updated