Introduction
As Large Language Models (LLMs) are increasingly integrated into high-stakes domains such as law and healthcare, understanding their subjective judgment becomes as critical as measuring their factual accuracy. While traditional benchmarks focus on objective truths, they often fail to capture how a model assigns importance to information—a process known as "salience detection." This paper introduces the Supernova Event Dataset, a novel benchmark designed to probe the latent "personalities" or "value lenses" of LLMs by requiring them to select and rank critical events from complex narratives.
Methodology
The researchers developed a framework that leverages Retrieval-Augmented Generation (RAG) to process long-form texts. The pipeline involves a two-stage prompting process where a model identifies and ranks five "critical events" that decisively altered a narrative's trajectory. To justify these rankings, models are encouraged to use counterfactual reasoning (e.g., "If this event had not occurred, would the outcome be different?").
To analyze these selections objectively, an "LLM-as-a-Judge" (specifically Qwen-2.5 14B) classifies the target model's choices into personality dimensions such as Ideological, Emotional, or Strategic. Finally, these traits are mapped into a semantic space using vector embeddings and visualized via Principal Component Analysis (PCA).
The Supernova Dataset
The benchmark consists of curated articles across four diverse categories:
- Biographies: 150 long-form Wikipedia entries of notable figures.
- Historical Events: 150 articles on global turning points from 1000 BCE to 2000 CE.
- Major News Events: 150 articles on significant events from the year 2000 to the present.
- Scientific Discoveries: 25 expanded articles covering Nobel-prize-winning breakthroughs.
An additional ablation study using a Movie Script Dataset (1,172 scripts) was conducted to verify the consistency of these personality traits across fictional narratives.
Key Insights: Inherent Value Lenses
A primary finding of this research is that personality in LLMs is inherent rather than merely simulated through role-playing. Even without explicit instructions to "act" a certain way, models exhibit stable behavioral patterns:
- Orca 2 consistently prioritizes Emotional and Interpersonal dynamics, focusing on relationship conflicts.
- Phi-4 acts as a Strategic Achiever, emphasizing immediate catalysts and professional milestones.
- Qwen 2.5 adopts a Systemic/Analytical approach, seeking underlying causes over surface-level events.
These "value lenses" also manifest in scientific analysis. For instance, OpenAI o3 tends to be causality-centric, while Gemini 2.5 Pro focuses on enablement (tools and validation), and Claude 3.7 Sonnet emphasizes conceptual synthesis and paradigm shifts.
Results and Model Separation
The study confirms that these "personalities" are distinct and measurable. PCA visualization shows a clear separation between models in the semantic space, indicating that their underlying value systems are unique. Furthermore, these tendencies remain consistent across different domains; a model's focus on strategy or emotion persists whether it is analyzing a financial crisis or a movie plot.
Limitations and Future Work
While the Supernova Event Dataset provides a powerful tool for interpreting LLM behavior, the authors note several limitations. The reliance on Wikipedia may introduce Western-centric biases, and the use of an LLM as a judge carries the risk of stylistic bias from the judge model itself. Future research should incorporate human validation to align AI-perceived personalities with human intuition and explore mechanistic interpretability to understand how these traits are represented within a model's internal weights.
Summary updated