AlienBench: A Benchmark of Structural Novelty in Language Model Outputs

Jonas Oppenlaender, Johanna M. SilvennoinenarXiv 2026 · arXiv

TL;DR

AlienBench is a new benchmark that uses cognitive psychology to measure "structural novelty"—the ability of Large Language Models (LLMs) to generate ideas that depart from familiar, real-world defaults. By asking models to describe alien creatures, the researchers found that open-weight models often outperform proprietary "frontier" models in creativity, and that current AI is significantly "weirder" than the average human.

What Problem Does This Paper Address?

Evaluating creativity in AI is notoriously difficult. Most current methods rely on "holistic" scores from human experts or other LLMs, which are subjective and lack a fixed reference point. If you ask an AI to be "creative," how do you objectively measure how far it strayed from the "ordinary"?

The authors tackle this by borrowing the structured imagination task from cognitive psychology. Decades of research show that when humans are asked to imagine something new (like an alien), they suffer from "design fixation," instinctively reusing Earth-based features like bilateral symmetry or legs for walking. AlienBench uses these "Earth defaults" as a fixed baseline to measure structural novelty: a model’s ability to propose fundamentally different biological mechanisms rather than just surface-level variations (e.g., a creature that moves via jet propulsion is structurally novel; a dog with six legs is not).

Methods

The core of the benchmark is the Ward Departure Score, which evaluates ten specific biological features:

Table 1: Biological features of the Ward Departure Score.

To ensure a robust evaluation, the researchers:

  1. Prompted 10 different LLMs (5 proprietary, 5 open-weight) to describe alien creatures under six different conditions, ranging from a simple baseline to constrained environments like "high gravity" or "ammonia oceans."
  2. Used a Panel of Judges: Three high-end LLMs (Claude 4.6, Gemini 3.1 Pro, and GPT-5.4) acted as a jury. They extracted the ten features from each description and labeled them as either "Earth-typical" (0) or a "departure" (1) based on strict boundary criteria.
  3. Aggregated Scores: Each model generated 180 descriptions in total. The final score is the mean number of departures (out of 10) across all generations.

Datasets

  • AlienBench: A newly introduced benchmark consisting of 1,800 model-generated descriptions, judge outputs, and scoring rubrics.
  • Human Baseline: The authors recruited 100 participants via Prolific to perform the same task. This provides a crucial "sanity check" for AI performance.
  • Model Pool: Includes frontier models like GPT-6 and Claude Opus 5.5 alongside open-weight contenders like Llama 4 Maverick and Ministral 3 14B.

Key Insights and Ideas

  • The "Weirdness" Gap: Every single LLM tested outperformed the human baseline. While humans averaged only 1.93 departures from Earth defaults, models scored between 3.28 and 6.55. AI seems less "fixated" on Earth biology than we are.
  • Open-Weight Superiority: In a surprising twist, open-weight models scored higher on average (5.40) than proprietary models (4.58). The authors suggest this might be due to proprietary models being more heavily "aligned" or filtered, which can sometimes lead to more conservative, "safe" outputs.
  • Structural vs. Surface Novelty: The benchmark successfully distinguishes between "more of the same" and "something different." By requiring a change in kind (e.g., moving from a centralized nervous system to a hive-mind), it filters out models that simply add more eyes or limbs to a standard animal template.
  • Prompt Sensitivity: Models are highly sensitive to framing. Simply telling a model to be "as different as possible" (Departure-primed) caused scores to skyrocket compared to the baseline prompt.

Key Results

The quantitative rankings revealed a clear winner in the "imagination" department:

  • Top Performer: Ministral 3 14B achieved the highest overall score of 6.55.
  • Proprietary Leader: Grok 4.7 was the most creative proprietary model with a score of 5.42.
  • Lowest Scorer: GPT-6 Luna trailed the pack at 3.28, scoring lower than every open-weight model tested.
Figure 1: Departure rate of each of the ten features, per model (larger area is better). The scale on each axis is 0–1. The left column shows the human baseline (blue) and the model with the highest overall Ward Departure Score.

As seen in the radar charts above, models like Ministral 3 14B show a much broader "coverage" of structural novelty across features like sensing and locomotion compared to the more constrained human baseline.

Limitations and Future Work

The authors are careful to note that high scores don't necessarily mean the AI is "thinking" creatively.

  • Memorized Tropes: LLMs are trained on vast amounts of science fiction. A model might not be "imagining" a novel metabolism so much as it is reciting a description of a Horta from Star Trek.
  • Elaboration Bias: The benchmark currently conflates novelty with length. A model that writes a 2,000-word essay has more opportunities to mention (and thus depart from) biological features than a model that writes a single paragraph.
  • Domain Specificity: AlienBench is limited to biology. Future work could apply this "departure from defaults" logic to other creative domains like architectural design or music theory.

Ultimately, AlienBench provides a much-needed objective yardstick for the "divergent thinking" capabilities of our increasingly creative AI assistants.

Summary updated

Abstract

Large language models (LLMs) are increasingly used for creative tasks such as writing, design, and idea generation. For these uses, it matters whether a model's ideas depart from familiar examples or stay close to them. Novelty is one standard measure of creativity, and current creativity evaluations ask human experts or LLM judges to rate how novel an output is. However, these ratings have no fixed reference for what counts as novel (or, in other terms, departure from the familiar). Measuring this departure requires a task with known familiar defaults. The structured imagination task from cognitive psychology is such a task. In this creativity test, people imagine a creature from another planet. The departure from features of Earth animals, such as bilateral symmetry and legs as mode of locomotion, is used as measure of creativity. We introduce AlienBench, a benchmark that adapts the structured imagination task to language models. Models describe alien creatures, and a panel of three LLM judges extracts ten biological features from the description and labels them as either Earth-typical or a departure. The sum of feature departures is the score of a description, from 0 to 10. Each model describes alien creatures under six prompt conditions that vary the framing of the task and the planetary environment. Each model generates 30 descriptions per condition (180 in total), and its overall score is the mean across these descriptions. We evaluate 10 language models, from frontier to open-weight models. Overall scores range from 3.28 (GPT-6 Luna) to 6.55 (Ministral 3 14B). Interestingly, open-weight models score higher than proprietary models in this task (5.40 vs. 4.58). Under the baseline prompt, the models depart from Earth defaults more often on average than 100 human participants on every feature. We release all prompts, generations, judge outputs, and scores.
BibTeX
@misc{oppenlaender2026alienbench,
  author = {Jonas Oppenlaender and Johanna M. Silvennoinen},
  title = {{AlienBench: A Benchmark of Structural Novelty in Language Model Outputs}},
  year = {2026},
  eprint = {2611.09094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09094},
}
Figure