LVLMs are Bad at Overhearing Human Referential Communication

Zhengxiang Wang, Weiling Li, Panagiotis Kaliosis, Owen Rambow, Susan BrennanEMNLP 2025

TL;DR

Large Vision-Language Models (LVLMs) struggle to understand human conversations where people refer to objects, even when they "overhear" multiple interactions. Despite advancements, LVLMs don't improve their understanding over time like humans do, indicating a significant gap in their ability to learn and adapt to conversational nuances.

What Problem Does This Paper Address?

When humans converse, they develop shared understandings and ways of referring to objects, often becoming more efficient over time. For embodied AI agents to be truly useful in human environments, they need to understand these referential communications, not just when directly addressed, but also when overhearing conversations. This paper investigates how well current LVLMs can perform this "overhearing" task, specifically focusing on whether they can learn and improve their understanding of referential expressions from repeated interactions between the same human partners. The challenge lies in grounding language to visual information within the dynamic context of ongoing, spontaneous human dialogue.

Methods

The researchers designed an experiment where LVLMs act as "overhearers" to a corpus of human-human conversations. These conversations involved pairs of participants playing a collaborative object-matching game over multiple rounds. The LVLMs were presented with transcripts of these conversations and images of the objects. Their task was to identify the correct sequence of objects based on the Director's descriptions.

The study evaluated seven state-of-the-art LVLMs. They tested the models' performance in two main ways:

  1. Single-round performance: Assessing how well LVLMs perform in a single overheard conversation.
  2. Performance evolution: Examining if LVLMs improve their accuracy as they "overhear" more conversations from the same human pair over multiple rounds of the same task. This directly addresses the question of whether LVLMs can learn and adapt to evolving conversational conventions.

To ensure robustness, they also conducted follow-up experiments varying factors like textual input formality, providing object summaries instead of full transcripts, and testing with different object ordering in the input images.

Datasets

The study utilizes a previously unpublished corpus of 80 human-to-human dialogues, totaling 27,902 words. This corpus was collected from 10 pairs of English-speaking undergraduates engaged in a collaborative object-matching task over repeated rounds. The task involved matching sets of pictures of dogs and baskets. The corpus is noted as being free from data contamination, making it a suitable testbed for evaluating LVLMs' ability to adapt to novel, spontaneous language.

Key Insights and Ideas

  • LVLMs are not natural overhearers: Unlike humans, LVLMs don't seem to learn or adapt their understanding of referential communication from repeated exposure to the same conversational partners. This suggests a fundamental difference in how they process and integrate information from ongoing dialogue.
  • "Late overhearers" are a challenge: The study mirrors human psycholinguistic findings where "late overhearers" (those who join a conversation partway through) perform worse. LVLMs also struggle to catch up when starting later in a conversation, failing to leverage prior context effectively.
  • Object descriptions are key: Follow-up experiments suggest that the difficulty might not lie in basic visual grounding of individual objects, but rather in integrating that grounding with the dynamic, evolving language of conversation. Providing object summaries significantly improved performance.
  • Proprietary models lead, but don't solve the problem: While proprietary LVLMs generally outperform open-weight models, even the best models fail to reach human-level performance in this overhearing task and do not show consistent improvement over time.

Key Results

  • Low accuracy: LVLMs achieved significantly lower accuracy than human participants (who achieved 100% accuracy). Even the best proprietary LVLMs (GPT-40, Gemini-2.0-Flash) averaged around 45-76% accuracy in single rounds, while open-weight models performed much worse.
  • No consistent improvement: Critically, all tested LVLMs, including GPT-40 and Gemini-2.0-Flash, failed to show a consistent performance improvement as they overheard more conversations from the same human pair over time. This was observed in at least 70% of human pairs.
  • Robustness issues: Performance varied significantly based on the specific human pair and the ordering of objects in the input image, highlighting a lack of robustness.
  • Object summaries help: Providing object summaries instead of full transcripts led to a substantial performance gain, indicating that the complexity of natural language descriptions in spontaneous conversation is a major hurdle.

Limitations and Future Work

The authors acknowledge the relatively small size of their corpus (80 dialogues) due to resource constraints. They also note the non-deterministic nature of LVLMs, which requires multiple runs for reliable evaluation. Future work could explore:

  • Mechanistic insights: Understanding why LVLMs fail to learn from overheard conversations.
  • Finetuning: Investigating if finetuning LVLMs on this type of data can improve performance.
  • More diverse tasks and models: Expanding the evaluation to a wider range of conversational scenarios and more LVLMs.
  • Human baselines: Conducting experiments with human participants acting as overhearers to directly compare performance.

Summary updated

BibTeX
@inproceedings{wang2025lvlms,
  author = {Zhengxiang Wang and Weiling Li and Panagiotis Kaliosis and Owen Rambow and Susan Brennan},
  title = {{LVLMs are Bad at Overhearing Human Referential Communication}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure