Who Holds the Pen? Caricature and Perspective in LLM Retellings of History

Lubna Zahan Lamia, Mabsur Fatin Bin Hossain, Md Mosaddek KhanEMNLP 2025

TL;DR

This paper investigates how Large Language Models (LLMs) portray historical events when prompted to adopt specific personas. They find that LLMs tend to exaggerate identity-specific traits, create conflicting narratives, and subtly alter grammatical agency, particularly for directly affected personas, raising concerns about the potential for LLMs to distort historical memory.

What Problem Does This Paper Address?

As LLMs become more sophisticated, they are increasingly used to simulate human perspectives and behaviors, including historical narratives. However, there's a critical need to understand how these models embed biases and asymmetries when framing, interpreting, and retelling the past. This paper addresses the question of whether LLMs produce genuine viewpoint variation or risk creating exaggerated, flattened, and potentially distorted historical accounts, especially when simulating personas with different relationships to an event.

Methods

The authors developed a framework to analyze LLM-generated historical narratives based on four core aspects:

  • Caricature Detection: This involves two components:
  • Individuation: Measuring how well LLM responses can be distinguished from neutral baselines when conditioned on a persona.
  • Exaggeration: Quantifying the degree to which persona-defining features are amplified beyond topical necessity, using semantic axes derived from word embeddings.
  • Framing Divergence (Contradiction): Using Natural Language Inference (NLI) models to measure the probability that one persona's narrative contradicts the other's for the same event.
  • Grammatical Agency (Passivity): Analyzing the ratio of passive to active voice constructions to understand how agency and responsibility are linguistically distributed, particularly how directly affected personas might use more passive constructions in institutional contexts.
  • Toxicity Bias Analysis: Employing the Detoxify model to assess seven dimensions of toxicity (e.g., identity attack, threat) to see if persona swapping introduces harmful language.

Datasets

The authors constructed a novel dataset of 197 historically grounded events spanning January 2000 to February 2025. Each event is paired with a directly affected persona (physically present, harmed, or in operational control) and an indirectly affected persona (observer, advocate, analyst). This resulted in 394 persona-based responses. The dataset was curated from publicly accessible APIs and credible news archives, with manual verification.

Key Insights and Ideas

  • Directly Affected Personas are More Exaggerated: LLMs amplify identity-specific language more when simulating directly affected personas, a key signal of caricature. This suggests LLMs lean into "identity-specific framing" more strongly for those closer to an event.
  • Conflicting Narratives Emerge, Especially in Contentious Domains: LLMs frequently generate contradictory interpretations of the same event, particularly in politically, militarily, and morally charged contexts. This highlights how LLMs can reflect deeply polarized historical interpretations.
  • Grammatical Passivity Varies Systematically: LLMs tend to use more passive constructions in institutional contexts and more active voice in personal ones. Directly affected personas in institutional settings might use passivity to deflect blame or depersonalize events.
  • Subtle Asymmetries, Not Overt Toxicity: The study found no significant increase in harmful or offensive language due to persona swapping. The distortion comes from subtle shifts in tone, stance, and emphasis, rather than overt toxicity.
  • Caricature is a Real Risk: The combination of individuation and exaggeration, especially for directly affected personas, indicates a systematic risk of LLMs producing caricatured historical accounts.

Key Results

  • Individuation: LLM responses were highly distinguishable from default outputs for both directly (92.3% accuracy) and indirectly (94.9% accuracy) affected personas, confirming persona-sensitive variation.
  • Exaggeration: Directly affected personas showed significantly higher exaggeration scores (mean difference of 0.074, Cohen's d = 0.47), indicating a moderate effect size.
  • Contradiction: A moderate effect size (Cohen's d = 0.43) was found for contradiction scores, suggesting systematic epistemic divergence between direct and indirect persona responses.
  • Grammatical Passivity: A small but statistically significant effect size (Cohen's d = 0.18) was observed for the delta passive ratio, indicating a systematic distortion in grammatical passivity.
  • Toxicity: No statistically significant differences in toxicity scores were found between direct and indirect persona responses across all seven dimensions.

Limitations and Future Work

Limitations:

  • Semantic Axes: The method for measuring exaggeration relies on context-specific semantic axes, which might conflate stylistic elaboration with intentional exaggeration, especially in emotional responses.
  • NLI Limitations: The NLI model for contradiction might overestimate conflict due to stylistic variation or miss divergence not framed as explicit semantic conflict.
  • Grammatical Parser: The parser might misclassify complex structures or miss implicit passivity conveyed through narrative stance.
  • Toxicity Detection: The toxicity model might miss more subtle narrative harms like ideological framing.
  • Dataset Scope: While diverse, the dataset doesn't capture every possible identity position or narrative setting.

Future Work:

  • Longitudinal Events: Modeling events that unfold over time to analyze shifts in blame or memory.
  • Fine-grained Personas: Incorporating more complex, intersecting personas (e.g., a refugee who is also an educator) to explore layered identities.
  • Human Evaluation: Validating automated metrics with human judgments on how identity, exaggeration, and bias are perceived.
  • Open-Source Models: Comparing proprietary LLMs with open-source alternatives to understand potential divergences.
  • Ethical Considerations: Emphasizing the need for human-in-the-loop auditing, transparent documentation, and careful consideration of how identity is encoded and expressed.

Summary updated

BibTeX
@inproceedings{lamia2025who,
  author = {Lubna Zahan Lamia and Mabsur Fatin Bin Hossain and Md Mosaddek Khan},
  title = {{Who Holds the Pen? Caricature and Perspective in LLM Retellings of History}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure