FLUID QA: A Multilingual Benchmark for Figurative Language Usage in Dialogue across English, Chinese, and Korean

Seoyoon Park, Hyeji Choi, Minseon Kim, Subin An, Xiaonan Wang, Gyuri Choi, Hansaem KimEMNLP 2025

TL;DR

Large language models (LLMs) are getting better at understanding figurative language, but they still struggle to use it appropriately in multi-turn conversations. This paper introduces FLUID QA, a new multilingual benchmark that tests LLMs' ability to select contextually appropriate figurative expressions in dialogue, revealing a significant gap between recognition and actual usage.

What Problem Does This Paper Address?

Figurative language, like metaphors and sarcasm, is crucial for nuanced communication, conveying emotion, stance, and social cues. While LLMs have shown promise in recognizing figurative language at the sentence level, their ability to use it coherently in dynamic, multi-turn dialogues remains largely untested and uncertain. Existing benchmarks often focus on isolated sentence classification, failing to capture the pragmatic and discourse-level understanding required for natural conversation. Furthermore, most research is English-centric, neglecting the cross-linguistic and culturally embedded nature of figurative language. This paper aims to bridge this gap by evaluating LLMs' figurative language usage in multilingual dialogue settings.

Methods

The authors introduce FLUID QA, a novel multilingual benchmark designed to assess figurative language usage in dialogue. Each item in FLUID QA presents a short multi-turn dialogue (3-4 turns) with a cloze-style prompt, requiring the LLM to choose the most pragmatically appropriate figurative expression from four candidates. This setup tests the LLM's ability to consider tone, speaker intent, and discourse-level appropriateness.

To aid in diagnosing usage-level failures, they also developed FLUTE-bi, a lightweight, sentence-level diagnostic task that focuses on figurative language recognition. This task uses parallel literal-figurative sentence pairs.

For multilingual construction, they employed a culturally adaptive prompting strategy using GPT-4o for Korean and Chinese translations, ensuring that idioms were replaced with culturally equivalent expressions to preserve semantic and pragmatic alignment. This approach was validated through pairwise preference comparisons by native speakers, which consistently favored culturally adapted translations.

Datasets

  • FLUID QA: The primary benchmark introduced in this paper. It consists of 800 QA items per language (English, Chinese, Korean), each featuring a multi-turn dialogue and multiple-choice figurative expression selection.
  • FLUTE-bi: A diagnostic dataset adapted from the existing FLUTE dataset. It contains 800 sentence pairs per language, each with a figurative and a literal version, used for sentence-level recognition tasks.

Key Insights and Ideas

  • The Recognition-Usage Gap is Significant: LLMs that perform well on sentence-level figurative recognition (FLUTE-bi) often fail dramatically on dialogue-level usage tasks (FLUID QA). This highlights that understanding how to use figurative language is much harder than simply identifying it.
  • Discourse-Level Reasoning is Key: Figurative language usage in dialogue requires more than just semantic knowledge; it demands an understanding of context, tone, and social intent, which current LLMs struggle with.
  • Language Dominance Shifts: While English generally leads in recognition tasks, the hierarchy shifts in dialogue usage, with English remaining strong, Korean in the middle, and Chinese becoming the weakest. This suggests discourse-level reasoning amplifies cross-lingual disparities.
  • Sarcasm is Particularly Challenging: Models perform worst on sarcasm, likely due to the need for implicit stance recognition and affective pragmatics, which go beyond surface-level semantic similarity. Idioms, conversely, are easiest due to their fixed syntactic forms.
  • Language Specialization Offers Limited Gains: While language-specific models show some advantage in their target languages, they don't consistently surpass proprietary multilingual models, indicating that specialization alone isn't enough for complex discourse-level tasks.

Key Results

  • Performance Drop: Across all models, performance on FLUID QA (usage) drops substantially compared to FLUTE-bi (recognition). For example, proprietary models like Claude 3.5 Sonnet achieve F1 scores around 0.80-0.85 on FLUTE-bi but drop to around 0.70-0.75 on FLUID QA.
  • Proprietary vs. Open-Source Divide: Proprietary models consistently outperform open-source models on both tasks, with the gap widening significantly in dialogue usage.
  • Cross-Lingual Disparities: In FLUID QA, English scores are highest, followed by Korean, and then Chinese, indicating that Chinese is the most vulnerable language for discourse-level figurative reasoning.
  • Category Performance: Sarcasm consistently yields the lowest F1 scores (below 0.25), while idioms are the easiest category. Metaphors and similes show moderate and unstable performance.
  • Confusion Patterns: Models frequently confuse similes with metaphors and often replace idioms with metaphors, suggesting a reliance on surface-level analogical similarity or lexical familiarity rather than discourse function. Sarcasm is often misinterpreted as idiomatic or literal.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Scope: The benchmark currently focuses on only four rhetorical categories and three languages.
  • Model Size: The evaluation did not systematically vary model size, leaving questions about the relationship between scale and figurative competence.
  • Few-Shot Settings: The evaluation was limited to zero- and few-shot settings, not exploring fine-tuning or instruction-tuning effects.

Future work could expand the category space, incorporate more cultural variations, and investigate adaptive methods for pragmatic alignment. The authors emphasize the need for future models to not only detect figurative language but also use it appropriately, reflecting speaker goals, emotional tone, and cultural context.

Summary updated

BibTeX
@inproceedings{park2025fluid,
  author = {Seoyoon Park and Hyeji Choi and Minseon Kim and Subin An and Xiaonan Wang and Gyuri Choi and Hansaem Kim},
  title = {{FLUID QA: A Multilingual Benchmark for Figurative Language Usage in Dialogue across English, Chinese, and Korean}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure