Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses

Sugyeong Eo, Heuiseok Lim

TL;DR

While large language models (LLMs) have become remarkably adept at parsing complex verbal text, they remain surprisingly blind to the unspoken elements of human conversation. This paper presents the first systematic evaluation of LLMs' ability to infer pragmatic meaning from purely non-verbal responses like silence, emojis, and physical gestures. The findings reveal a massive performance gap: while LLMs achieve near-human accuracy on verbal responses, their accuracy plummets by up to 60 percentage points when forced to rely solely on non-verbal cues.

What Problem Does This Paper Address?

Human communication is inherently pragmatic; we routinely say one thing but mean another, or say nothing at all to convey a message (e.g., remaining silent after being asked "Am I disturbing you?" implies an uncomfortable "Yes"). While recent NLP research shows that LLMs are growing increasingly competent at understanding indirect verbal speech, real-world interactions frequently rely on standalone non-verbal cues to convey intent.

This paper addresses a critical question: Can LLMs recognize implicit communicative intent when verbal utterances are entirely absent? To answer this, the authors systematically evaluate how well models interpret three distinct categories of standalone non-verbal responses: silence, facial expressions, and bodily movements.

Methods

The authors design a multiple-choice probing framework where models are given a dialogue context ending in a non-verbal response and must select the option that best interprets the underlying intent. To rigorously test the models' pragmatic boundaries, the authors construct four specific types of "distractors" (incorrect options):

  1. Misinterpretations: Reversing or inaccurately inferring the meaning.
  2. Literal explanations: Describing surface-level behavior without attributing pragmatic meaning.
  3. Lexically coherent but semantically divergent responses: Options with surface-level similarity but zero contextual relevance.
  4. Abstract explanations: Vague, oversimplified descriptions that fail to address the core topic.

The evaluation spans 13 instruction-tuned models across 6 prominent model families (ranging from 3B to over 100B parameters) in a zero-shot setting. Additionally, the authors leverage advanced analytical techniques like the logit lens to track internal layer-wise prediction dynamics, and evaluate mitigation strategies such as Chain-of-Thought (CoT) prompting and few-shot in-context learning (ICL).

Datasets

The authors introduce a new, high-quality evaluation dataset consisting of 50 self-collected situational prompts representing everyday social interactions. To isolate the impact of non-verbal communication, each scenario is systematically varied across four response formats within identical contexts:

  • Silence: Represented textually as '...'
  • Facial Expressions: Represented via Unicode emojis (e.g., 😅, 🙄).
  • Movements: Represented as descriptive actions enclosed in parentheses (e.g., (Points to the broken laptop.)).
  • Verbal Responses: Used as a baseline control group where the response is indirect but explicitly spoken (e.g., "Oh, my legs hurt" to decline help moving).

The dataset underwent strict quality control involving automated LLM grading (via GPT-4o and GPT-4.1-mini) and human cross-checking to ensure clarity, correctness, and difficulty.

Key Insights and Ideas

  • Silence is the hardest puzzle: LLMs struggle most when there is an absolute absence of overt signals. Silence carries high "communicative richness" but provides the least explicit data, causing models to fail drastically compared to humans.
  • The "Abstract" Trap: When LLMs fail, they don't just misinterpret the cue; they default to oversimplified, vague, and abstract descriptions (e.g., stating only that "A asks a question and B responds with a behavior") rather than engaging in deeper inferential reasoning.
  • CoT can backfire: Interestingly, Chain-of-Thought prompting often degrades performance for non-verbal reasoning. Models like GPT-4o frequently default to overly literal interpretations under CoT, leading to a performance drop of over 0.3. Conversely, few-shot in-context learning consistently helps larger models internalize the relational patterns.
  • Deeper layers do the heavy lifting: Logit lens analysis revealed that confidence scores for the correct options remain uniformly low in the early layers and only begin to emerge in the middle-to-later layers of the networks, where fine-grained semantic and contextual representations are formed.

Key Results

  • The Verbal vs. Non-Verbal Gap: While humans maintain high accuracy across both verbal (0.99) and non-verbal (0.91) settings, LLMs show a massive drop. For instance, 3B-Llama drops from 0.96 accuracy on verbal text down to 0.37 on non-verbal text.
  • Category Performance: Across all tested models, performance was highest for movements (avg. 0.67 accuracy), followed by facial expressions (0.51), and lowest for silence (0.45).
Evaluation of LLM performance across three non-verbal response categories
  • Scale Matters: Larger models (32B and 70B+) significantly outperform smaller models (3B to 14B), indicating that scaling up parameter size enhances a model's capacity to infer intentions from non-verbal cues.
Distribution of average performance across LLMs of varying parameter scales
  • Error Distribution: Abstract explanations accounted for over half of all errors across all categories (57% for silence, 73% for facial expressions, 69% for movements), showing a clear failure to capture implicit intent.
Distribution of errors across four distractor types

Limitations and Future Work

  • English-Centric: The evaluation dataset is entirely in English and assumes cross-cultural universality of pragmatic inferences, ignoring language-specific or culture-specific non-verbal nuances. The authors suggest expanding this to a multilingual setup.
  • Textual Modality Only: Non-verbal behaviors are represented purely as text (emojis, descriptions, ellipses) rather than actual visual or auditory modalities. Incorporating multimodal inputs (video/audio) in future work will be essential to truly enhance the applicability of LLMs to real-world human-AI interactions.

Summary updated

Abstract

Although large language models (LLMs) have shown considerable progress in pragmatic language understanding, prior research has focused mainly on their comprehension of verbal behavior. Nonetheless, non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings. In this work, we present the first systematic evaluation of LLMs' ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. We explore three research questions: (1) Can LLMs recognize indirect intent conveyed through non-verbal responses? (2) When and how do LLMs fail to capture non-verbal intent? (3) How can we improve LLMs' ability to interpret non-verbal intent?. Through the evaluation, we observe that LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60% points compared to verbal ones. Further extensive analysis reveals a behavioral pattern in LLMs' interpretations of non-verbal behavior and demonstrates that in-context learning facilitates pragmatic inference.
BibTeX
@inproceedings{eo2026unveiling,
  author = {Sugyeong Eo and Heuiseok Lim},
  title = {{Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2101},
}

Loading related content...

Figure