MuseScorer: Idea Originality Scoring At Scale

Ali Sarosh Bangash, Krish Veera, Ishfat Abrar Islam, Raiyan Abdul BatenEMNLP 2025

TL;DR

This paper introduces MUSESCORER, a novel system that automates the scoring of idea originality by measuring its statistical infrequency within a population. It leverages Large Language Models (LLMs) with an "LLM-as-a-judge" framework and external retrieval to achieve psychometrically validated, human-aligned originality assessment at scale, overcoming limitations of previous manual and computational methods.

What Problem Does This Paper Address?

Assessing creativity often involves evaluating two dimensions: intrinsic qualities and extrinsic statistical infrequency. While computational methods have advanced for intrinsic qualities, scoring originality based on how rare an idea is within a population remains a significant challenge. This process traditionally requires manual "bucketing" of similar ideas, which is subjective, labor-intensive, error-prone, and doesn't scale well. Existing computational approaches struggle with the nuances of semantic similarity versus true rephrasing, the fat-tailed distribution of idea frequencies, and the ever-growing number of idea buckets. This lack of a scalable, objective, and validated method hinders creativity research.

Methods

MUSESCORER employs a fully automated, psychometrically validated system for frequency-based originality scoring. The core innovation lies in its "LLM-as-a-judge" framework combined with externally orchestrated retrieval. For each new idea, the system first retrieves semantically similar prior "idea buckets" using K-NN search. Then, a Large Language Model (LLM) is zero-shot prompted to determine if the new idea fits an existing bucket or forms a new one. This approach mimics human bucketing granularity and intent sensitivity. The system uses a dynamic codebook that is updated as new ideas arrive. They experimented with different LLM variants (Llama3.3, Qwen3, Phi4), sentence embedding models, and prompting strategies (vanilla vs. Chain-of-Thought), finding a specific combination to be optimal.

Datasets

The study utilizes five Alternative Uses Test (AUT) datasets:

  • socialmuse24: Used for primary dataset acquisition and establishing criterion validity. It includes human annotations for idea bucketing and Creativity Quotient scores.
  • beaty18, silvia17, beaty12, mohr16: These publicly available datasets are used for assessing convergent and external validity. They provide various creativity-related ratings and psychological measures.

Key Insights and Ideas

  • LLMs as nuanced judges: The "LLM-as-a-judge" paradigm, when combined with retrieval, can effectively replicate the subjective and intent-sensitive nature of human idea bucketing, which is crucial for originality scoring.
  • Fat-tailed distributions are key: Human-annotated idea bucket sizes exhibit fat-tailed distributions. Any automated system must account for this to achieve human-like performance, a challenge for traditional clustering methods.
  • Threshold metric shines: Among the frequency-based originality metrics explored, the "threshold" metric demonstrated the most robust performance and best alignment with human judgments across various validation measures.
  • CoT prompting improves interpretability: Chain-of-Thought (CoT) prompting not only improved performance but also provided justifications for bucketing decisions, enhancing the system's transparency.

Key Results

MUSESCORER demonstrated strong performance across multiple dimensions:

  • Idea-level bucketing: Achieved high agreement with human annotators, with an Adjusted Mutual Information (AMI) score of 0.59, significantly outperforming clustering-based baselines (best baseline AMI of 0.39).
  • Participant-level scoring: Showed strong correlations with human originality scores, with Pearson's r = 0.89 and ICC(3,1) = 0.88 for the best configuration (Llama3.3 with CoT prompting).
  • Convergent validity: Normalized originality scores correlated highly with Creativity Quotient (CQ) and human creative quality judgments.
  • External validity: Showed systematic correlations with metaphor generation quality, openness to experience, and creative identity/self-efficacy.

Limitations and Future Work

  • Demographic fairness and accessibility: Variations in language use across cultural backgrounds could bias bucketing judgments, especially in non-English contexts.
  • Generalizability: Validation is currently limited to AUT-style tasks; generalization to other creative domains (e.g., design, visual arts) needs further investigation.
  • Prompt sensitivity: The system's performance can be sensitive to prompt length and phrasing, requiring careful prompt engineering.
  • Efficiency: While the current approach prioritizes stability, processing ideas one by one limits throughput. Future work could explore batching or multi-step reasoning for increased efficiency.
  • Heuristic cutoffs: The effectiveness of the threshold metric relies on heuristic cutoffs that require further validation.
  • Hallucination risk: As with all LLM-based systems, there's a risk of misassigning ideas to incorrect buckets, though the system's strong alignment with human annotations suggests reasonable reliability within its scope.

Summary updated

BibTeX
@inproceedings{bangash2025musescorer,
  author = {Ali Sarosh Bangash and Krish Veera and Ishfat Abrar Islam and Raiyan Abdul Baten},
  title = {{MuseScorer: Idea Originality Scoring At Scale}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure