TL;DR
AlienBench is a new benchmark that uses cognitive psychology to measure "structural novelty"—the ability of Large Language Models (LLMs) to generate ideas that depart from familiar, real-world defaults. By asking models to describe alien creatures, the researchers found that open-weight models often outperform proprietary "frontier" models in creativity, and that current AI is significantly "weirder" than the average human.
What Problem Does This Paper Address?
Evaluating creativity in AI is notoriously difficult. Most current methods rely on "holistic" scores from human experts or other LLMs, which are subjective and lack a fixed reference point. If you ask an AI to be "creative," how do you objectively measure how far it strayed from the "ordinary"?
The authors tackle this by borrowing the structured imagination task from cognitive psychology. Decades of research show that when humans are asked to imagine something new (like an alien), they suffer from "design fixation," instinctively reusing Earth-based features like bilateral symmetry or legs for walking. AlienBench uses these "Earth defaults" as a fixed baseline to measure structural novelty: a model’s ability to propose fundamentally different biological mechanisms rather than just surface-level variations (e.g., a creature that moves via jet propulsion is structurally novel; a dog with six legs is not).
Methods
The core of the benchmark is the Ward Departure Score, which evaluates ten specific biological features:
To ensure a robust evaluation, the researchers:
- Prompted 10 different LLMs (5 proprietary, 5 open-weight) to describe alien creatures under six different conditions, ranging from a simple baseline to constrained environments like "high gravity" or "ammonia oceans."
- Used a Panel of Judges: Three high-end LLMs (Claude 4.6, Gemini 3.1 Pro, and GPT-5.4) acted as a jury. They extracted the ten features from each description and labeled them as either "Earth-typical" (0) or a "departure" (1) based on strict boundary criteria.
- Aggregated Scores: Each model generated 180 descriptions in total. The final score is the mean number of departures (out of 10) across all generations.
Datasets
- AlienBench: A newly introduced benchmark consisting of 1,800 model-generated descriptions, judge outputs, and scoring rubrics.
- Human Baseline: The authors recruited 100 participants via Prolific to perform the same task. This provides a crucial "sanity check" for AI performance.
- Model Pool: Includes frontier models like GPT-6 and Claude Opus 5.5 alongside open-weight contenders like Llama 4 Maverick and Ministral 3 14B.
Key Insights and Ideas
- The "Weirdness" Gap: Every single LLM tested outperformed the human baseline. While humans averaged only 1.93 departures from Earth defaults, models scored between 3.28 and 6.55. AI seems less "fixated" on Earth biology than we are.
- Open-Weight Superiority: In a surprising twist, open-weight models scored higher on average (5.40) than proprietary models (4.58). The authors suggest this might be due to proprietary models being more heavily "aligned" or filtered, which can sometimes lead to more conservative, "safe" outputs.
- Structural vs. Surface Novelty: The benchmark successfully distinguishes between "more of the same" and "something different." By requiring a change in kind (e.g., moving from a centralized nervous system to a hive-mind), it filters out models that simply add more eyes or limbs to a standard animal template.
- Prompt Sensitivity: Models are highly sensitive to framing. Simply telling a model to be "as different as possible" (Departure-primed) caused scores to skyrocket compared to the baseline prompt.
Key Results
The quantitative rankings revealed a clear winner in the "imagination" department:
- Top Performer: Ministral 3 14B achieved the highest overall score of 6.55.
- Proprietary Leader: Grok 4.7 was the most creative proprietary model with a score of 5.42.
- Lowest Scorer: GPT-6 Luna trailed the pack at 3.28, scoring lower than every open-weight model tested.
As seen in the radar charts above, models like Ministral 3 14B show a much broader "coverage" of structural novelty across features like sensing and locomotion compared to the more constrained human baseline.
Limitations and Future Work
The authors are careful to note that high scores don't necessarily mean the AI is "thinking" creatively.
- Memorized Tropes: LLMs are trained on vast amounts of science fiction. A model might not be "imagining" a novel metabolism so much as it is reciting a description of a Horta from Star Trek.
- Elaboration Bias: The benchmark currently conflates novelty with length. A model that writes a 2,000-word essay has more opportunities to mention (and thus depart from) biological features than a model that writes a single paragraph.
- Domain Specificity: AlienBench is limited to biology. Future work could apply this "departure from defaults" logic to other creative domains like architectural design or music theory.
Ultimately, AlienBench provides a much-needed objective yardstick for the "divergent thinking" capabilities of our increasingly creative AI assistants.
Summary updated