Evaluating the values and biases of Large Language Models (LLMs) is a critical challenge in AI safety and alignment. Traditional methods often rely on closed-form tests like the Political Compass Test (PCT), where models are asked to "Strongly Agree" or "Disagree" with specific propositions. However, these tests are notoriously brittle; a model's stance can shift dramatically based on minor prompt variations or the specific persona it is instructed to adopt. More importantly, these tests fail to capture the why—the underlying reasoning or justifications for a model's choice.
This research introduces the concept of "tropes"—recurring, semantically consistent justifications found in open-ended text. By identifying these tropes, researchers can move beyond simple labels to understand the latent values and logic that persist within a model, even when it is steered toward different political personas.
Methodology: The Trope Extraction Pipeline
To systematically study these justifications, the authors developed a robust pipeline to extract tropes from a massive dataset of model responses:
- Massive Prompting: The researchers generated over 156,000 responses across six different LLMs (including Llama 2, Llama 3, Mistral, and others). They used 62 PCT propositions and 420 prompt variations, mixing demographic personas (e.g., age, gender, nationality) with various instruction styles.
- Trope Extraction:
- Clustering: Responses were decomposed into individual sentences and converted into semantic embeddings using S-BERT.
- DBSCAN: This density-based clustering algorithm was applied to group sentences that expressed essentially the same argument.
- Distillation: An LLM (GPT-4o-mini) was used to filter these clusters, ensuring they contained substantive justifications rather than generic filler text.
- Analysis: The researchers then compared the prevalence and similarity of these tropes across different models and personas using metrics like Jaccard similarity and Total Variation Distance (TVD).
Key Findings: Steerability and the "Left-Leaning" Default
The study revealed significant insights into how LLMs handle political personas. One of the most striking findings was the "left-leaning" default of many models. When prompted with "far-left" personas, models showed much less variation in their answers compared to when they were prompted with "far-right" ones. This suggests that the models' pre-training data or alignment processes may have ingrained a default state more aligned with left-leaning viewpoints, making those personas easier to simulate consistently.
Furthermore, the degree of "steerability"—how much a model's political placement changes based on its assigned persona—varied significantly between models.
As seen in the figure above, models like Llama 3 exhibit a wide spread across the political compass when prompted with different demographic and political personas. In contrast, other models like OLMo were found to be more resistant to such steering.
The Open-vs-Closed Gap
A critical discovery was the massive disparity between a model's choices in a multiple-choice format (closed-form) versus its justifications in open-ended writing. For instance, when adopting "far-right" personas, Llama 3's open-ended justifications disagreed with its own forced-choice selections 90% of the time. This "Open-vs-Closed Gap" highlights the limitations of relying solely on multiple-choice benchmarks to evaluate model values.
Conclusion and Future Directions
The identification of 584 distinct tropes across the dataset provides a new lens through which to view LLM behavior. The high internal consistency of these trope clusters (92%) suggests they are reliable indicators of a model's underlying logic. The researchers also found that model lineage matters; models sharing architectures or training data exhibited similar trope "fingerprints."
While the study is limited by its Western-centric focus and the size of the models tested (under 13B parameters), it sets a new precedent for evaluating AI alignment. Future safety work should focus not just on the final answer a model provides, but on the tropes it uses to justify its positions, as these reveal the deeper, latent biases that closed-form tests miss.
Summary updated