TL;DR
This paper introduces PACE, a novel and efficient metric for evaluating the creativity of Large Language Models (LLMs) by measuring their ability to generate distant word associations. PACE correlates strongly with existing LLM leaderboards and human evaluations, demonstrating its effectiveness in capturing creative potential without the risks of data contamination.
What Problem Does This Paper Address?
Evaluating the creative capabilities of LLMs is a critical but challenging area of AI research. Existing methods often suffer from data contamination, where LLMs might have been trained on evaluation datasets, compromising the reliability of results. Furthermore, human evaluation, while valuable, is subjective, time-consuming, and difficult to scale. This paper aims to bridge this gap by proposing a simple, efficient, and scalable metric that mimics human associative creativity assessment while mitigating the aforementioned issues.
Methods
The core of this paper is the PACE (Parallel Association Chain Evaluation) metric. It's inspired by human associative creativity tests and involves two main stages:
- Seed Word Association: For a given seed word, the LLM generates three distinct "secondary" seed words.
- Parallel Association Chains: For each primary and secondary seed pair, the LLM generates a 20-word association chain. Crucially, each word in the chain is associated only with the immediately preceding word, and the generation is done independently for each chain to encourage diverse pathways.
The "creativity" is then quantified by measuring the association distance within these chains. This distance reflects how "distant" or unconventional the associations are, moving beyond simple co-occurrence. The metric uses FastText embeddings to calculate semantic distances between words. The overall creativity score for a model is an average of these association distances across many seed words.
Datasets
- Seed Words: 110 seed words were selected from the Intercontinental Dictionary Series (IDS), chosen for their semantic diversity and frequency across various domains.
- Human Data: The study leverages existing human data from Gray et al. (2019) for comparison, which includes a general population sample and a professional group (actors).
- LLM Leaderboards: The paper uses rankings from established LLM leaderboards such as Chatbot Arena (Overall and Creative Writing), MMLU-Pro, LiveBench, and EQ-Bench for validation.
Key Insights and Ideas
- Creativity as Distant Association: The paper strongly advocates for the idea that creativity, particularly in language, is deeply rooted in the ability to form non-obvious, distant associations between concepts.
- LLMs Mimic, But Don't Surpass Professionals: High-performing LLMs can match the associative creativity of average humans, but professional humans still maintain an edge, especially in terms of diversity of association types.
- Concreteness Trend: Both humans and LLMs tend to produce associations that become less concrete as the chain progresses. However, LLMs generally exhibit higher baseline concreteness.
- Diversity Matters: While LLMs are improving, they still tend to produce more homogeneous responses compared to humans, particularly in lexical diversity (Type-Token Ratio). Professionals show even greater diversity in their association patterns.
- PACE's Effectiveness: The proposed PACE metric shows a strong correlation with existing benchmarks, suggesting it effectively captures a crucial aspect of LLM creativity.
Key Results
- Strong Correlation: PACE demonstrated a significant Spearman rank correlation of 0.739 (p < 0.001) with the Chatbot Arena Creative Writing rankings, outperforming correlations with general performance leaderboards.
- Human vs. LLM Performance: High-performing LLMs showed comparable associative creativity to general human groups, but professional humans consistently outperformed them.
- Model Differentiation: PACE effectively differentiates between model variants within the same series (e.g., different versions of DeepSeek or Qwen), showing sensitivity to model updates and sizes.
- Lexical Diversity Gap: LLMs consistently exhibit lower Type-Token Ratios (TTR) than humans, indicating less lexical diversity in their generated associations.
Limitations and Future Work
- English-Centric: The current evaluation is limited to English due to the use of English seed words and leaderboards. Future work could explore multilingual evaluations.
- Sample Size for Leaderboards: While the primary analyses use sufficient models, some comparisons with other leaderboards are closer to the minimum sample size threshold, potentially limiting robustness.
- Model Selection Constraints: Reliance on external leaderboards inherently limits the selection of models to those represented on those platforms.
- Future Work: The authors suggest exploring the impact of different chain lengths and seed word counts on correlation, as well as investigating the influence of model parameters and architectures on associative creativity across various semantic categories. They also highlight the potential for PACE to be used for fine-tuning LLMs to enhance their creative capabilities.
Summary updated