Evaluating the Creativity of LLMs in Persian Literary Text Generation

Armin Tourajmehr, Mohammad Reza Modarres, Yadollah YaghoobzadehEMNLP 2025

TL;DR

This paper introduces a novel framework for evaluating the creativity of Large Language Models (LLMs) in generating Persian literary texts, adapting the Torrance Tests of Creative Thinking (TTCT) for cultural relevance. They also introduce CPers, a new dataset of Persian literary texts, and find that while LLMs show promise, their creativity doesn't yet match human nuance and diversity, particularly in the use of literary devices.

What Problem Does This Paper Address?

The field of creative text generation by LLMs has largely focused on English, leaving a significant gap in understanding their capabilities in other languages, especially those with rich literary traditions but fewer digital resources, like Persian. Furthermore, existing methods for evaluating creativity often fall short when applied to culturally specific literary contexts, failing to capture the nuances of human expression. This paper aims to bridge this gap by developing a culturally sensitive evaluation framework and dataset for Persian literary text generation and assessing the performance of several state-of-the-art LLMs.

Methods

The core of the methodology involves adapting the Torrance Tests of Creative Thinking (TTCT) to evaluate Persian literary texts. The TTCT's four dimensions—originality, fluency, flexibility, and elaboration—are assessed through culturally tailored questions. To address the limitations of traditional fluency metrics in short Persian texts, the authors redefine fluency to encompass grammatical accuracy, naturalness for native speakers, and appropriateness for literary or conversational contexts. They also introduce an LLM-as-a-judge approach, using Claude 3.7 Sonnet, and validate its reliability against human judgments using Intraclass Correlation Coefficients (ICC). Additionally, the study analyzes the use of four common literary devices (simile, metaphor, antithesis, hyperbole) to understand their impact on perceived creativity.

Datasets

The paper introduces CPers, a newly curated dataset of 4,371 Persian literary texts spanning 20 diverse emotional and cultural topics. This dataset is designed to be a benchmark for evaluating creativity in low-resource languages. A subset of 100 texts from CPers (Human texts) and 100 generated texts from GPT-3.5 (Model texts) were used for human annotation and evaluation.

Key Insights and Ideas

  • Cultural Nuance is Key: Evaluating creativity in LLMs requires more than just general metrics; it demands a deep understanding of cultural context and linguistic nuances, especially for literary applications.
  • LLMs Still Lag in True Creativity: While LLMs can generate coherent and grammatically sound texts, they often rely on conventional expressions and struggle with genuine novelty and diverse stylistic choices compared to human writers.
  • Literary Devices Matter: The balanced and nuanced application of literary devices, not just their quantity, significantly influences perceived creativity. Overreliance on common devices like similes can hinder originality.
  • LLM-as-a-Judge is Promising, but Imperfect: Claude 3.7 Sonnet showed strong alignment with human judgments for creativity evaluation, but LLMs still struggle with nuanced tasks like distinguishing between specific figures of speech.
  • Reasoning-Oriented Models Show Potential: Models trained with reasoning capabilities (like DeepSeek-R1) demonstrated stronger performance in elaboration and flexibility, suggesting that structured thinking can enhance creative output.

Key Results

  • DeepSeek-R1 achieved the highest overall creativity score, excelling in flexibility and elaboration.
  • GPT-4.1 showed strong fluency and readability but lower originality, often using conventional expressions.
  • Qwen2.5-VL-32B-Instruct had high originality but struggled with fluency and elaboration.
  • Human annotators showed strong agreement (high ICC scores) on both human-written and model-generated texts, validating the evaluation framework.
  • Claude 3.7 Sonnet demonstrated significantly better alignment with human judgments than GPT-4o when acting as an LLM judge.
  • LLMs tended to overuse similes and metaphors, while human texts showed a more balanced and integrated use of various literary devices.

Limitations and Future Work

The authors acknowledge several limitations:

  • The evaluation questions were designed by the authors, who, while fluent, are not formally trained psychologists or literary scholars. Future work could benefit from interdisciplinary collaboration.
  • The analysis focused on 100 samples and five themes, providing a limited view of the broader dataset. Creativity might be more evident in longer narratives.
  • The study used a zero-shot prompting strategy. Exploring few-shot or chain-of-thought prompting could reveal more about LLM capabilities.
  • The cross-lingual aspect is limited to Persian. Future work should explore how LLM creativity varies across different languages and cultures.

Future work suggested includes increasing the number of annotators, improving LLM judges, testing diverse prompting strategies, and conducting cross-lingual comparisons.

Summary updated

BibTeX
@inproceedings{tourajmehr2025evaluating,
  author = {Armin Tourajmehr and Mohammad Reza Modarres and Yadollah Yaghoobzadeh},
  title = {{Evaluating the Creativity of LLMs in Persian Literary Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure