Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Intent Resolution in LLMs

Akhila Yerukola, Saujas Vaduguru, Daniel Fried, Maarten SaparXiv 2024 · DOI · arXiv

Abstract

While Large Language Models (LLMs) demonstrate increasing fluency, they frequently overlook the unspoken nuances of human conversation. This paper introduces a generative framework to evaluate how well LLMs resolve non-literal intent—such as irony or metaphors. The study finds that models struggle significantly to generate pragmatically appropriate responses, often defaulting to literal interpretations even when they possess the underlying knowledge of the speaker's meaning.

Introduction: The Pragmatic Gap

Human communication is rarely strictly literal. We employ irony to signal disagreement, metaphors to simplify complex concepts, and indirect speech for politeness (e.g., saying "It's cold in here" to imply "Please close the window"). This field of study, known as pragmatics, is essential for natural interaction.

Prior research primarily evaluated LLM pragmatic abilities using discriminative tasks, such as multiple-choice questions identifying a speaker's meaning. However, identifying a definition does not guarantee a model can apply that understanding in a generative context. This paper shifts the focus to generative evaluation: can a model produce a response reflecting a speaker's true intent rather than just their literal words?

Methodology: Counterfactual Evaluation

The authors propose a framework that evaluates an LLM's generated response ($U^N_2$) against two "counterfactual" reference responses:

  1. The True Intent Response ($U^T_2$): The response expected if the speaker had spoken directly and literally.
  2. The Literal Interpretation Response ($U^L_2$): The response expected if the non-literal statement were taken at face value.
The counterfactual evaluation framework comparing literal vs. true intent responses.

A model is deemed "pragmatically accurate" if its generated response is more similar to the True Intent reference than the Literal reference. Similarity was measured using human annotators, GPT-4 as a contextual evaluator, and Llama-3-based text embeddings. The study focused on four phenomena: Indirect Speech, Irony, Maxims of Conversation (e.g., abrupt subject changes), and Metaphor.

Key Results and the "Knowing vs. Doing" Gap

The evaluation revealed a stark disconnect between a model's ability to detect intent and its ability to act on it. On average, state-of-the-art open-source models (like Llama-2 and Mistral) achieved only 50-55% accuracy in generative settings—barely exceeding random chance.

Generative vs. Discriminative accuracy across different pragmatic phenomena for various LLMs.

Key findings include:

  • The Literal Gravity Well: LLMs exhibit a strong "literal bias." For instance, when faced with hyperbole like "I have a million books," models often respond to the literal quantity ("That's an ambitious list") rather than the sentiment ("You must love reading!").
  • Maxims are the Hardest: Models struggled most with "flouted maxims," often performing worse than random chance. They failed to recognize social cues, such as changing the subject to be polite, and instead forced the conversation back to the literal topic.
  • Scaling and Prompting: While larger models generally performed better, the generative-discriminative gap persisted. Chain-of-Thought (CoT) prompting provided only modest gains. Even providing "Oracle" intent (perfect information about the speaker's meaning) did not lead to 100% accuracy, suggesting the challenge lies in the generation process itself.

Conclusion and Future Directions

The study highlights a significant limitation in current LLMs: the inability to bridge the gap between understanding and utilizing pragmatic intent. Future work should explore longer conversational contexts, a wider array of linguistic phenomena (like idioms and proverbs), and the influence of social dynamics and emotional states. Furthermore, researchers must consider the "factuality trade-off," as training models to be less literal might impact their performance in technical or medical contexts where literal accuracy is paramount.

Summary updated

Abstract

Humans often express their communicative intents indirectly or non-literally, which requires their interlocutors -- human or AI -- to understand beyond the literal meaning of words. While most existing work has focused on discriminative evaluations, we present a new approach to generatively evaluate large language models' (LLMs') intention understanding by examining their responses to non-literal utterances. Ideally, an LLM should respond in line with the true intention of a non-literal utterance, not its literal interpretation. Our findings show that LLMs struggle to generate pragmatically relevant responses to non-literal language, achieving only 50-55% accuracy on average. While explicitly providing oracle intentions significantly improves performance (e.g., 75% for Mistral-Instruct), this still indicates challenges in leveraging given intentions to produce appropriate responses. Using chain-of-thought to make models spell out intentions yields much smaller gains (60% for Mistral-Instruct). These findings suggest that LLMs are not yet effective pragmatic interlocutors, highlighting the need for better approaches for modeling intentions and utilizing them for pragmatic generation.
BibTeX
@misc{yerukola2024is,
  author = {Akhila Yerukola and Saujas Vaduguru and Daniel Fried and Maarten Sap},
  title = {{Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Intent Resolution in LLMs}},
  year = {2024},
  eprint = {2405.08760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.08760},
}
Figure