Testing Language Creativity of Large Language Models and Humans

Anca Dinu, Andra-Maria FlorescuNAACL 2025

TL;DR

This paper introduces a novel, integrated test to directly compare the language creativity of Large Language Models (LLMs) and humans. The study reveals LLMs exhibit slightly superior performance, particularly in figurative language, while humans tend to be more pragmatic. This work provides a valuable benchmark for evaluating AI creativity and highlights the nuanced differences between human and machine linguistic innovation.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has sparked intense interest in their creative capabilities. While LLMs excel at various tasks, understanding and evaluating their "creativity" remains a significant challenge. Existing research often focuses on specific aspects of creativity or uses fragmented testing methods. This paper addresses the need for a comprehensive, integrated benchmark that can directly compare LLM and human language creativity across multiple dimensions, moving beyond task-specific performance to assess genuine linguistic innovation. The authors aim to bridge the gap in standardized evaluation methods for AI creativity, a field that is rapidly evolving and requires robust assessment tools.

Methods

The core of this research is a custom-designed language creativity test, administered to both LLMs and humans. This test is structured to incorporate both divergent (generating many ideas) and convergent (selecting the best ideas) thinking, drawing inspiration from established psychological creativity tests. It comprises twelve distinct tasks, each with multiple items, designed to assess various facets of linguistic creativity, including figurative language, word formation, and semantic novelty.

The evaluation of the generated responses was primarily automated using a software called "Open Creativity Scoring with LLMs" (OCSAI). This tool, trained on a fine-tuned set of LLMs, correlates highly with human judgment for creativity assessment. To ensure reliability, a portion of the data was manually scored by one of the authors, demonstrating a strong inter-annotator agreement. The authors also employed computational analysis techniques such as sentiment analysis, clusterization (using DistilROBERTa embeddings and PCA), and binary classification (using transformer models like DistilROBERTa, T5, and BERT) to further investigate the differences and similarities between human and LLM responses.

Datasets

The study collected a dataset of language creativity answers from 15 different LLMs and 20 human participants. The LLMs included a range of popular models such as ChatGPT, Claude, Gemini, Llama, and Mixtral, accessed through various platforms. The human participants were non-native English speakers with a B2-C2 English proficiency level, averaging 26 years old, and primarily university students. The dataset comprises a total of 4095 responses, totaling 17384 words, with human responses slightly outnumbering LLM responses in terms of both quantity and word count. This dataset represents a newly introduced benchmark for evaluating language creativity in both humans and LLMs.

Key Insights and Ideas

  • LLMs are surprisingly creative: The study demonstrates that LLMs possess remarkable creative abilities, often exhibiting human-like linguistic innovation across the spectrum from F-creativity (rule-based) to E-creativity (norm-breaking).
  • Figurative language is an LLM strength: LLMs particularly excel at tasks involving figurative speech, such as metaphors and novel expressions, often producing more poetic and elaborate responses than humans.
  • Humans lean towards pragmatism: While LLMs might be more "poetic," human responses, though sometimes less elaborate, often demonstrate a greater degree of pragmatism and a more personal touch, incorporating irony, humor, and cultural references.
  • Automated evaluation is feasible and reliable: The use of OCSAI for scoring proved to be a robust and reproducible method, correlating well with human judgment, which is crucial for scaling creativity assessments.
  • Subtle differences persist: Despite LLMs' strong performance, computational analysis (clustering and classification) reveals that human and LLM answers, while showing similarities, also exhibit distinct patterns, suggesting that current LLMs, while mimicking creativity, may not be entirely indistinguishable from human output.

Key Results

LLMs achieved slightly higher overall creativity scores (mean 0.49) compared to humans (mean 0.45), with a statistically significant difference (p-value = 0.0008). This advantage was observed across most creativity criteria: originality, flexibility, and elaboration. LLMs also outperformed humans in most of the individual tasks, with notable exceptions in "Rename" and "Euphemism/Dysphemism" tasks. The standard deviation for LLM scores was higher than for humans, indicating greater variability in their performance. Sentiment analysis revealed that while LLMs produced some negative sentiment responses (as expected by task requirements), they tended to be less negative than humans, potentially due to built-in filters. Binary classification models could distinguish between human and LLM answers with an accuracy of up to 0.80, suggesting identifiable differences in their linguistic output.

Limitations and Future Work

The authors acknowledge that the study primarily involved non-native English speakers, and future work should include native speakers to see if results differ. They also suggest that further qualitative analysis is needed to fully understand the nature of the differences and similarities between human and machine answers. The paper highlights the ongoing debate about whether LLM creativity is "genuine" or a sophisticated form of pattern matching from training data. Future research could focus on gathering more data, including native speaker responses, and conducting more in-depth qualitative analyses to further explore these nuances.

Summary updated

BibTeX
@inproceedings{dinu2025testing,
  author = {Anca Dinu and Andra-Maria Florescu},
  title = {{Testing Language Creativity of Large Language Models and Humans}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure