TL;DR
This paper introduces MUSESCORER, a novel system that automates the scoring of idea originality by measuring its statistical infrequency within a population. It leverages Large Language Models (LLMs) with an "LLM-as-a-judge" framework and external retrieval to achieve psychometrically validated, human-aligned originality assessment at scale, overcoming limitations of previous manual and computational methods.
What Problem Does This Paper Address?
Assessing creativity often involves evaluating two dimensions: intrinsic qualities and extrinsic statistical infrequency. While computational methods have advanced for intrinsic qualities, scoring originality based on how rare an idea is within a population remains a significant challenge. This process traditionally requires manual "bucketing" of similar ideas, which is subjective, labor-intensive, error-prone, and doesn't scale well. Existing computational approaches struggle with the nuances of semantic similarity versus true rephrasing, the fat-tailed distribution of idea frequencies, and the ever-growing number of idea buckets. This lack of a scalable, objective, and validated method hinders creativity research.
Methods
MUSESCORER employs a fully automated, psychometrically validated system for frequency-based originality scoring. The core innovation lies in its "LLM-as-a-judge" framework combined with externally orchestrated retrieval. For each new idea, the system first retrieves semantically similar prior "idea buckets" using K-NN search. Then, a Large Language Model (LLM) is zero-shot prompted to determine if the new idea fits an existing bucket or forms a new one. This approach mimics human bucketing granularity and intent sensitivity. The system uses a dynamic codebook that is updated as new ideas arrive. They experimented with different LLM variants (Llama3.3, Qwen3, Phi4), sentence embedding models, and prompting strategies (vanilla vs. Chain-of-Thought), finding a specific combination to be optimal.
Datasets
The study utilizes five Alternative Uses Test (AUT) datasets:
- socialmuse24: Used for primary dataset acquisition and establishing criterion validity. It includes human annotations for idea bucketing and Creativity Quotient scores.
- beaty18, silvia17, beaty12, mohr16: These publicly available datasets are used for assessing convergent and external validity. They provide various creativity-related ratings and psychological measures.
Key Insights and Ideas
- LLMs as nuanced judges: The "LLM-as-a-judge" paradigm, when combined with retrieval, can effectively replicate the subjective and intent-sensitive nature of human idea bucketing, which is crucial for originality scoring.
- Fat-tailed distributions are key: Human-annotated idea bucket sizes exhibit fat-tailed distributions. Any automated system must account for this to achieve human-like performance, a challenge for traditional clustering methods.
- Threshold metric shines: Among the frequency-based originality metrics explored, the "threshold" metric demonstrated the most robust performance and best alignment with human judgments across various validation measures.
- CoT prompting improves interpretability: Chain-of-Thought (CoT) prompting not only improved performance but also provided justifications for bucketing decisions, enhancing the system's transparency.
Key Results
MUSESCORER demonstrated strong performance across multiple dimensions:
- Idea-level bucketing: Achieved high agreement with human annotators, with an Adjusted Mutual Information (AMI) score of 0.59, significantly outperforming clustering-based baselines (best baseline AMI of 0.39).
- Participant-level scoring: Showed strong correlations with human originality scores, with Pearson's r = 0.89 and ICC(3,1) = 0.88 for the best configuration (Llama3.3 with CoT prompting).
- Convergent validity: Normalized originality scores correlated highly with Creativity Quotient (CQ) and human creative quality judgments.
- External validity: Showed systematic correlations with metaphor generation quality, openness to experience, and creative identity/self-efficacy.
Limitations and Future Work
- Demographic fairness and accessibility: Variations in language use across cultural backgrounds could bias bucketing judgments, especially in non-English contexts.
- Generalizability: Validation is currently limited to AUT-style tasks; generalization to other creative domains (e.g., design, visual arts) needs further investigation.
- Prompt sensitivity: The system's performance can be sensitive to prompt length and phrasing, requiring careful prompt engineering.
- Efficiency: While the current approach prioritizes stability, processing ideas one by one limits throughput. Future work could explore batching or multi-step reasoning for increased efficiency.
- Heuristic cutoffs: The effectiveness of the threshold metric relies on heuristic cutoffs that require further validation.
- Hallucination risk: As with all LLM-based systems, there's a risk of misassigning ideas to incorrect buckets, though the system's strong alignment with human annotations suggests reasonable reliability within its scope.
Summary updated