TL;DR
PILAPS is a new bilingual (English/Mandarin) benchmark designed to test whether Large Language Models (LLMs) can "read between the lines" across different cultures and conversational settings. The study reveals that while frontier models perform well on simple, purpose-written tests, they struggle significantly with real-world dialogue and remain far more literal-minded than human participants.
What Problem Does This Paper Address?
Understanding language isn't just about knowing what words mean; it’s about knowing what a speaker intends. This is called pragmatic competence. For example, if someone says "The kitchen is a bit warm," they might be making a factual observation, or they might be politely asking you to open a window.
While recent studies suggest LLMs are getting better at pragmatics in English, it’s unclear if this success translates to other languages or more complex, natural conversations. Most existing benchmarks are English-only and rely on short, artificial examples. Furthermore, pragmatics is deeply tied to culture—a literal translation of a joke or a social cue often loses its meaning entirely. The authors address this by creating a benchmark that accounts for cultural nuances and tests models on both "constructed" scenarios and "dialogue-derived" data from real TV scripts.
Methods
The researchers developed PILAPS (Pragmatic Inference across LAnguages, Phenomena and Settings), which covers eight distinct pragmatic phenomena, including irony, humor, and indirect speech.
A key innovation is their cultural adaptation protocol. Instead of simple translation, they used a two-stage process:
- LLM-Assisted Adaptation: Models like GPT-4o were prompted to preserve the pragmatic function of an item while adapting cultural references (e.g., changing "Black Friday" to "Double Eleven" for a Chinese context).
- Human Revision: Native speakers manually validated and refined these translations to ensure they felt natural and preserved the original intent.
The team evaluated eight different LLMs (four of Chinese origin, four of English origin) using three prompting strategies: Vanilla (direct question), Chain-of-Thought (CoT) (asking the model to think step-by-step), and Theory-of-Mind (ToM) (asking the model to reason about the speaker's mental state).
Datasets
PILAPS consists of 1,886 multiple-choice items split into two main components:
- Constructed Component: Purpose-written short scenarios adapted from the PragMega dataset.
- Dialogue-derived Component: Real-world conversational data. This includes a "brief" set and an "extended" set sourced from popular sitcoms like The Big Bang Theory (English) and My Own Swordsman (Mandarin).
Key Insights and Ideas
- The "Literal Bias": Models are significantly more prone to literal interpretations than humans, especially in complex dialogue. They often fall for "distractor" answers that have high lexical overlap with the prompt but miss the underlying point.
- Context Matters: There is a sharp performance drop when moving from simple, purpose-written scenarios to extended dialogues. Real conversations require integrating information across multiple turns, which remains a major hurdle for current models.
- Origin Asymmetry: A model's "home" language matters. English-origin models (like Llama) saw a significant performance dip in Chinese, whereas Chinese-origin models (like Qwen) actually performed slightly better in Chinese than in English.
- ToM vs. CoT: Interestingly, Theory-of-Mind prompting helped reduce literal errors by forcing the model to consider intent. Conversely, Chain-of-Thought sometimes increased literal errors, suggesting that "thinking more" doesn't always lead to "understanding better" if the model's reasoning path is flawed.
Key Results
- Top Performer: Gemini-3.5-Flash achieved the highest overall accuracy in both languages (~93%).
- The Dialogue Gap: Across all models, accuracy dropped from ~73% on constructed items to ~55% on extended dialogue in English.
- Error Profiles: In the constructed component, literal responses accounted for nearly 79% of errors in English. As shown in the chart below, while models improve on extended dialogue, they still rely heavily on literal or associative cues compared to the more nuanced errors humans might make.
Limitations and Future Work
The authors acknowledge that PILAPS is currently limited to two high-resource languages (English and Mandarin). The dialogue data is also drawn from a limited number of TV series, which might introduce specific stylistic biases.
Future work should expand to low-resource languages and move beyond multiple-choice formats toward interactive evaluations, where models must not only identify intent but also generate appropriate, pragmatically sound responses in real-time conversation.
Summary updated