Abstract
While Large Language Models (LLMs) demonstrate increasing fluency, they frequently overlook the unspoken nuances of human conversation. This paper introduces a generative framework to evaluate how well LLMs resolve non-literal intent—such as irony or metaphors. The study finds that models struggle significantly to generate pragmatically appropriate responses, often defaulting to literal interpretations even when they possess the underlying knowledge of the speaker's meaning.
Introduction: The Pragmatic Gap
Human communication is rarely strictly literal. We employ irony to signal disagreement, metaphors to simplify complex concepts, and indirect speech for politeness (e.g., saying "It's cold in here" to imply "Please close the window"). This field of study, known as pragmatics, is essential for natural interaction.
Prior research primarily evaluated LLM pragmatic abilities using discriminative tasks, such as multiple-choice questions identifying a speaker's meaning. However, identifying a definition does not guarantee a model can apply that understanding in a generative context. This paper shifts the focus to generative evaluation: can a model produce a response reflecting a speaker's true intent rather than just their literal words?
Methodology: Counterfactual Evaluation
The authors propose a framework that evaluates an LLM's generated response ($U^N_2$) against two "counterfactual" reference responses:
- The True Intent Response ($U^T_2$): The response expected if the speaker had spoken directly and literally.
- The Literal Interpretation Response ($U^L_2$): The response expected if the non-literal statement were taken at face value.
A model is deemed "pragmatically accurate" if its generated response is more similar to the True Intent reference than the Literal reference. Similarity was measured using human annotators, GPT-4 as a contextual evaluator, and Llama-3-based text embeddings. The study focused on four phenomena: Indirect Speech, Irony, Maxims of Conversation (e.g., abrupt subject changes), and Metaphor.
Key Results and the "Knowing vs. Doing" Gap
The evaluation revealed a stark disconnect between a model's ability to detect intent and its ability to act on it. On average, state-of-the-art open-source models (like Llama-2 and Mistral) achieved only 50-55% accuracy in generative settings—barely exceeding random chance.
Key findings include:
- The Literal Gravity Well: LLMs exhibit a strong "literal bias." For instance, when faced with hyperbole like "I have a million books," models often respond to the literal quantity ("That's an ambitious list") rather than the sentiment ("You must love reading!").
- Maxims are the Hardest: Models struggled most with "flouted maxims," often performing worse than random chance. They failed to recognize social cues, such as changing the subject to be polite, and instead forced the conversation back to the literal topic.
- Scaling and Prompting: While larger models generally performed better, the generative-discriminative gap persisted. Chain-of-Thought (CoT) prompting provided only modest gains. Even providing "Oracle" intent (perfect information about the speaker's meaning) did not lead to 100% accuracy, suggesting the challenge lies in the generation process itself.
Conclusion and Future Directions
The study highlights a significant limitation in current LLMs: the inability to bridge the gap between understanding and utilizing pragmatic intent. Future work should explore longer conversational contexts, a wider array of linguistic phenomena (like idioms and proverbs), and the influence of social dynamics and emotional states. Furthermore, researchers must consider the "factuality trade-off," as training models to be less literal might impact their performance in technical or medical contexts where literal accuracy is paramount.
Summary updated