Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis

Pranav Agarwal, Ioana CiucaarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) are increasingly integrated into high-stakes domains such as law and healthcare, understanding their subjective judgment becomes as critical as measuring their factual accuracy. While traditional benchmarks focus on objective truths, they often fail to capture how a model assigns importance to information—a process known as "salience detection." This paper introduces the Supernova Event Dataset, a novel benchmark designed to probe the latent "personalities" or "value lenses" of LLMs by requiring them to select and rank critical events from complex narratives.

Methodology

The researchers developed a framework that leverages Retrieval-Augmented Generation (RAG) to process long-form texts. The pipeline involves a two-stage prompting process where a model identifies and ranks five "critical events" that decisively altered a narrative's trajectory. To justify these rankings, models are encouraged to use counterfactual reasoning (e.g., "If this event had not occurred, would the outcome be different?").

To analyze these selections objectively, an "LLM-as-a-Judge" (specifically Qwen-2.5 14B) classifies the target model's choices into personality dimensions such as Ideological, Emotional, or Strategic. Finally, these traits are mapped into a semantic space using vector embeddings and visualized via Principal Component Analysis (PCA).

Overview of the Supernova Event Dataset pipeline.

The Supernova Dataset

The benchmark consists of curated articles across four diverse categories:

  • Biographies: 150 long-form Wikipedia entries of notable figures.
  • Historical Events: 150 articles on global turning points from 1000 BCE to 2000 CE.
  • Major News Events: 150 articles on significant events from the year 2000 to the present.
  • Scientific Discoveries: 25 expanded articles covering Nobel-prize-winning breakthroughs.

An additional ablation study using a Movie Script Dataset (1,172 scripts) was conducted to verify the consistency of these personality traits across fictional narratives.

Key Insights: Inherent Value Lenses

A primary finding of this research is that personality in LLMs is inherent rather than merely simulated through role-playing. Even without explicit instructions to "act" a certain way, models exhibit stable behavioral patterns:

  • Orca 2 consistently prioritizes Emotional and Interpersonal dynamics, focusing on relationship conflicts.
  • Phi-4 acts as a Strategic Achiever, emphasizing immediate catalysts and professional milestones.
  • Qwen 2.5 adopts a Systemic/Analytical approach, seeking underlying causes over surface-level events.

These "value lenses" also manifest in scientific analysis. For instance, OpenAI o3 tends to be causality-centric, while Gemini 2.5 Pro focuses on enablement (tools and validation), and Claude 3.7 Sonnet emphasizes conceptual synthesis and paradigm shifts.

Results and Model Separation

The study confirms that these "personalities" are distinct and measurable. PCA visualization shows a clear separation between models in the semantic space, indicating that their underlying value systems are unique. Furthermore, these tendencies remain consistent across different domains; a model's focus on strategy or emotion persists whether it is analyzing a financial crisis or a movie plot.

PCA visualization of model personality clusters.

Limitations and Future Work

While the Supernova Event Dataset provides a powerful tool for interpreting LLM behavior, the authors note several limitations. The reliance on Wikipedia may introduce Western-centric biases, and the use of an LLM as a judge carries the risk of stylistic bias from the judge model itself. Future research should incorporate human validation to align AI-perceived personalities with human intuition and explore mechanistic interpretability to understand how these traits are represented within a model's internal weights.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly integrated into everyday applications. As their influence grows, understanding their decision making and underlying personality becomes essential. In this work, we interpret model personality using our proposed Supernova Event Dataset, a novel dataset with diverse articles spanning biographies, historical events, news, and scientific discoveries. We use this dataset to benchmark LLMs on extracting and ranking key events from text, a subjective and complex challenge that requires reasoning over long-range context and modeling causal chains. We evaluate small models like Phi-4, Orca 2, and Qwen 2.5, and large, stronger models such as Claude 3.7, Gemini 2.5, and OpenAI o3, and propose a framework where another LLM acts as a judge to infer each model's personality based on its selection and classification of events. Our analysis shows distinct personality traits: for instance, Orca 2 demonstrates emotional reasoning focusing on interpersonal dynamics, while Qwen 2.5 displays a more strategic, analytical style. When analyzing scientific discovery events, Claude Sonnet 3.7 emphasizes conceptual framing, Gemini 2.5 Pro prioritizes empirical validation, and o3 favors step-by-step causal reasoning. This analysis improves model interpretability, making them user-friendly for a wide range of diverse applications. Project Page - https://www.supernova-event.ai/
BibTeX
@misc{agarwal2025supernova,
  author = {Pranav Agarwal and Ioana Ciuca},
  title = {{Supernova Event Dataset: Interpreting Large Language Models' Personality through Critical Event Analysis}},
  year = {2025},
  eprint = {2506.12189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.12189},
}
Figure