TL;DR
This paper introduces a novel framework and datasets for evaluating Large Language Models' (LLMs) ability to understand and reason with Persian literary allusions. The study reveals a significant gap: while LLMs excel at factual recall of allusions, they struggle to activate this knowledge in open-ended, figurative contexts, suggesting limitations in spontaneous cultural reasoning rather than knowledge deficits.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly capable of processing and generating human-like text. However, a critical aspect of human language understanding is the ability to grasp cultural nuances, including the use of allusions – indirect references to culturally significant entities. Recognizing an allusion requires more than just literal comprehension; it demands accessing and applying stored cultural knowledge in novel contexts. This paper tackles the challenge of evaluating this specific capability in LLMs, particularly within the rich tradition of Persian literature, which is replete with symbolic and indirect references. Existing LLM evaluation methods often focus on factual recall or multiple-choice formats, which may not adequately probe the spontaneous activation of cultural knowledge required for figurative language. The authors aim to bridge this gap by developing methods to assess how well LLMs can truly reason with cultural knowledge, not just remember it.
Methods
The researchers propose a dual framework for evaluating LLM allusion recognition. This framework integrates three key components:
- Knowledge Assessment: An open-ended task where LLMs are prompted to explain specific allusions, testing their foundational knowledge of their origin, meaning, and narrative components.
- Multiple-Choice (MC) Recognition: A task where LLMs are presented with a text and several options, one of which is the correct allusion. This tests their ability to discriminate between similar allusions and recognize them when presented with choices.
- Open-Ended Recognition: A more challenging task where LLMs must detect, identify, and explain the significance of an allusion within a given text without any pre-provided options. This directly probes spontaneous knowledge activation and contextual reasoning.
The evaluation protocol is designed to disentangle knowledge possession from knowledge activation, aiming to pinpoint whether LLM failures stem from missing information or difficulties in applying it.
Datasets
The paper introduces two novel datasets:
- PersPoems Dataset: This dataset consists of 200 annotated Persian poetry couplets containing allusions. Each allusion is annotated with its poet, theme, entities, and a description. The dataset is sourced from the Ganjoor corpus and covers six themes: mythical-historical, religious, mystical, Quranic, romantic, and other.
- LLM-Generated Dataset: This dataset comprises 75 novel, allusive texts generated by Claude 3.7 Sonnet. These texts incorporate allusions in out-of-distribution contexts, specifically designed to test allusion recognition beyond potential memorization of training data.
Key Insights and Ideas
- The "Knowledge vs. Activation" Gap: The most striking finding is the significant performance drop LLMs exhibit when moving from multiple-choice tasks to open-ended allusion recognition. This suggests that LLMs possess substantial factual knowledge about allusions but struggle to spontaneously activate this knowledge without explicit cues.
- Reasoning-Optimized Models Generalize Better: LLMs that have been specifically fine-tuned for reasoning (e.g., through Reinforcement Learning) demonstrate more consistent performance across both datasets, indicating a better integration of cultural reasoning capabilities.
- Distillation Can Harm Cultural Reasoning: The study observed that distilled models (variants of larger models trained to be smaller) can experience a substantial performance drop in cultural reasoning tasks, suggesting that distillation might inadvertently remove crucial knowledge or reasoning abilities.
- Surface-Level Matching in MC Tasks: In multiple-choice settings, some LLMs with limited knowledge can achieve surprisingly high accuracy by relying on surface-level lexical or semantic similarities between the text and the options, rather than genuine comprehension. This highlights a potential pitfall in MC-based evaluations.
Key Results
The evaluation involved 11 LLMs (6 open-source, 5 closed-source).
- Knowledge Assessment: Most LLMs demonstrated strong foundational knowledge, with seven models exceeding 90% accuracy.
- Multiple-Choice Recognition: LLMs performed very well on MC tasks, with accuracies ranging from 74.0% to 93.5% on the PersPoems dataset and up to 96.0% on the LLM-Generated dataset.
- Open-Ended Recognition: Performance dropped significantly in open-ended tasks. Claude 3.5-Sonnet achieved the highest at 80.5% on PersPoems, while GPT-4.1 led at 74.6% on the LLM-Generated dataset. The gap between MC and open-ended performance was substantial, with some models dropping over 50%.
- Cross-Dataset Stability: RL-trained models showed remarkable consistency between the PersPoems and LLM-Generated datasets, while non-RL models exhibited more significant drops, reinforcing the benefit of RL for generalization.
Limitations and Future Work
The authors acknowledge several limitations:
- The study focuses solely on allusions, excluding other figurative devices like metaphors, irony, or symbolism.
- The evaluation is limited to Persian cultural and literary allusions, potentially missing broader cross-cultural patterns.
- The use of a single LLM for generating the allusive text might introduce biases.
- A more comprehensive taxonomy of failure modes could further enhance understanding.
Future research directions suggested include:
- Investigating whether the observed knowledge-activation gap extends to other figurative language forms.
- Conducting cross-linguistic comparative studies.
- Employing diverse generation strategies for creating test data.
- Developing detailed error pattern analyses to improve LLM reasoning for figurative language.
- Exploring the viability of LLM-as-Judge for automated evaluation, particularly when provided with ground truth.
Summary updated