Evaluating Taxonomy Free Character Role Labeling (TF-CRL) in News Stories using Large Language Models

David G Hobson, Derek Ruths, Andrew PiperEMNLP 2025

TL;DR

This paper introduces Taxonomy-Free Character Role Labeling (TF-CRL), a novel method for assigning descriptive, open-ended role labels to characters in news stories. By leveraging Large Language Models (LLMs), TF-CRL offers a more nuanced and flexible approach than traditional fixed taxonomies, enabling richer narrative analysis and cross-domain comparisons.

What Problem Does This Paper Address?

Understanding the roles characters play in narratives is crucial for comprehending how events are framed and how meanings are constructed. Existing methods often rely on predefined taxonomies (like Hero-Villain-Victim), which can be too rigid, limiting the ability to capture the full spectrum of character functions, especially in diverse domains like news. This lack of nuance hinders comparative analysis and the discovery of novel narrative patterns. The authors aim to address this by developing a "taxonomy-free" approach that allows for more flexible and descriptive role labeling.

Methods

The core of the paper is the Taxonomy-Free Character Role Labeling (TF-CRL) task. This involves assigning open-ended, short noun phrase labels to characters based on their functional role within a news story. The labels are compositional, consisting of a primary label (e.g., "Leader") and an optional modifier (e.g., "Resilient"), creating labels like "Resilient Leader."

The authors leverage Large Language Models (LLMs) like GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 for this task, employing zero-shot generation. They also explore different prompt pipelines, including concept induction, summarization, and direct zero-shot labeling. For evaluation, they use human preference rankings and ratings based on four criteria: Faithfulness, Relevance, Informativeness, and Generalizability.

Datasets

  • Global News dataset: Used for validation, comprising 120 news articles from four outlets (Al Jazeera English, The Times of India, BBC News, ABC News) across ten categories.
  • Media Frames Corpus (MFC): Used for the application case study, providing articles from US-based outlets on topics like immigration, climate change, and same-sex marriage.
  • Russia-Ukraine and Technology datasets: Also used in the application case study, sourced from the Global News dataset.

Key Insights and Ideas

  • Beyond Fixed Taxonomies: The TF-CRL approach demonstrates that open-ended, compositional labels can provide richer insights than rigid, predefined role categories.
  • LLMs Excel at Nuance: LLMs, particularly GPT-4o, are highly capable of generating these nuanced labels, often outperforming human annotators in terms of faithfulness and relevance.
  • Discovering Latent Taxonomies: TF-CRL can reveal emergent character role typologies within specific domains that might not fit traditional frameworks. For example, technology news might highlight "investor-innovator" relationships rather than just heroes and villains.
  • Cross-Narrative Comparisons: The method enables meaningful comparisons of character roles across different news domains, highlighting similarities and differences in how similar roles are portrayed.
  • Role Entropy as a Measure of Nuance: The concept of "role entropy" can quantify the diversity and consistency of character portrayals within a narrative or across multiple articles.

Key Results

  • LLM Performance: LLMs consistently outperformed human annotators across all evaluation criteria (Faithfulness, Relevance, Informativeness, Generalizability), with GPT-4o showing the strongest performance.
  • Faithfulness and Relevance: LLMs achieved high scores in faithfulness and relevance, indicating their ability to accurately capture character roles as depicted in the text.
  • Informativeness and Generalizability: LLMs also scored well on informativeness and generalizability, suggesting their labels provide useful distinctions and can be applied across different contexts.
  • HVV Classification: GPT-4o achieved an average F1 score of 88% when classifying characters into Hero, Villain, or Victim categories, demonstrating the utility of TF-CRL in enhancing existing typologies.
  • Cross-Narrative Similarity: The Russia-Ukraine dataset showed the least similarity with other domains, reflecting its distinct focus on conflict-related roles, while Immigration, Climate, and SSM showed higher similarity.

Limitations and Future Work

  • Label Space Size: The open-vocabulary nature of TF-CRL results in a very large number of unique labels, posing challenges for analysis and requiring further research into effective clustering and label space reduction.
  • Language Support: The current validation is limited to English articles, and expanding to other languages is crucial for capturing cultural nuances.
  • Character Relationships: The current focus is on individual character roles, and future work could extend the task to explicitly capture relationships between characters.
  • LLM Biases: Reliance on LLMs raises concerns about potential biases in assigned roles. Future work should involve targeted validation across diverse news sources and annotators to ensure accuracy and mitigate bias.
  • Computational Cost: The use of large LLMs for this task can be computationally expensive, and exploring the capabilities of smaller models is an area for future research.

Summary updated

BibTeX
@inproceedings{hobson2025evaluating,
  author = {David G Hobson and Derek Ruths and Andrew Piper},
  title = {{Evaluating Taxonomy Free Character Role Labeling (TF-CRL) in News Stories using Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure