TL;DR
This paper investigates how well Large Language Models (LLMs) can identify and label topics in narrative texts, comparing their performance to human annotators and traditional topic modeling techniques. The findings suggest LLMs are highly effective, even outperforming humans in fiction and matching them in news, offering a promising new tool for digital humanities research.
What Problem Does This Paper Address?
Topic modeling has been a cornerstone of digital humanities for years, helping researchers understand large collections of text by identifying underlying themes. Traditional methods like Latent Dirichlet Allocation (LDA) have limitations: topics can be ambiguous, sensitive to parameter choices, and oversimplify text by treating it as a "bag of words." Furthermore, LDA struggles with short texts and can be challenging to interpret, especially for narrative content which relies heavily on implicit context, figurative language, and complex temporal structures. This paper aims to bridge this gap by evaluating LLMs for narrative topic labeling, extending prior work on fact-based documents to the richer, more complex domain of stories. The core question is whether LLMs can effectively label narrative topics as well as, or even better than, humans and existing methods.
Methods
The researchers employed a multi-pronged approach:
- LLM Evaluation: They tested four different LLMs (GPT-4o, Llama3:8B, Llama3.1:8B, and Gemma2:9B) using a zero-shot prompting framework. This means the LLMs were given a passage and asked to identify three topics in descending order of generality, without prior fine-tuning on the specific dataset.
- Human Annotation: Two undergraduate humanities students were hired as annotators. They were provided with a clear definition of "topic" and a transcript of a ChatGPT discussion to guide their understanding. They also provided three topic labels per passage.
- LDA Baseline: Latent Dirichlet Allocation (LDA) was used with two different numbers of topics (k=20 and k=60) to establish a baseline. For each passage, the top three most probable topics were selected to align with the LLM output.
- Quantitative Validation: A ranked voting survey was conducted with 200 crowdworkers. Participants were presented with a passage, a topic definition, and five possible labels: four from different LLMs and one human annotation. They ranked these labels from best to worst.
- Qualitative Assessment: The study also included a qualitative analysis, examining specific examples of human and LLM-generated labels to understand preferences and nuances.
- LLM Topic Aggregation: A significant challenge with LLMs is the sheer volume of unique labels. To address this, the researchers developed a topic aggregation method using GPT-4o to consolidate similar and synonymous labels into a more manageable set of general topics.
Datasets
The study utilized two primary datasets:
- Fiction Dataset: A curated collection of approximately 700 open-access novels published in the nineteenth century, divided into 500-word chunks.
- Fact-Based Dataset (News): 6,722 news articles from the Global News Dataset, representing four publications (ABC News, Al Jazeera English, BBC News, and The Times of India). Articles were used whole due to their average length.
For the quantitative validation, 50 passages/articles were sampled from each dataset. For the case study, a larger sample of 25,000 passages from the novel data was used.
Key Insights and Ideas
- LLMs Excel in Fiction: Surprisingly, LLMs not only matched human performance in labeling news articles but outperformed humans in labeling fictional narratives. This suggests LLMs are adept at capturing the nuanced, implicit, and figurative language common in stories.
- Zero-Shot is Sufficient: The direct, zero-shot prompting approach for LLMs proved effective, performing on par with human annotations without the need for complex pre-processing or fine-tuning. This significantly streamlines the topic labeling process.
- The "Long Tail" Problem: LLMs generate a vast number of unique topic labels, many of which are semantically similar or redundant. The proposed aggregation method helps manage this, but it's an area ripe for further research.
- Specificity Matters (Sometimes): For news, readers often preferred more specific labels. However, for fiction, the preference was less clear-cut, with general labels sometimes being preferred. This highlights the context-dependent nature of label utility.
- LLMs Offer Thematic Richness: Compared to LDA, LLM-generated topics are more detailed, informative, and capture a wider range of thematic diversity without introducing unintelligible noise. This allows for more nuanced analysis of cultural and historical shifts.
Key Results
- Quantitative Validation: In fiction, Gemma2 received the best average rank (2.25), while humans received the worst (3.23). In news, GPT-4o performed best (2.40), with Llama3 and Llama3.1 performing similarly to humans (2.84 and 2.79 respectively). Statistically significant differences were found between human and LLM labels for fiction, but not for news.
- Human-LLM Similarity: Human annotations matched at least one LLM output in 50% of news cases and 72% of novel cases.
- LLM-LLM Consistency: At least two LLMs generated identical outputs for 98% of news passages and 94% of novel passages, indicating high internal consistency among LLMs.
- Case Study: In a comparison of 19th-century British novels, LLM-generated topics like "combat," "revenge," and "travel" provided more insightful distinctions for the pre-1850 period than LDA topics like "seemed" or "conduct." Similarly, emerging themes of "civil war" and "slavery" were more clearly identified by LLMs in the later period.
Limitations and Future Work
- Model Generalizability: The study compared a limited set of LLMs. Future work could explore a wider range of models, including smaller, fine-tuned ones, and investigate how different genres might affect results.
- Human Annotator Bias: The lower preference for human labels in fiction might be due to the specific annotators or biases in the survey participants. A larger sample of human respondents could provide further insights.
- Topic Aggregation: The current aggregation method is a step towards managing LLM label diversity but has limitations. Future research should explore alternative clustering and aggregation techniques.
- Computational Cost: LLMs, especially larger ones, require significant computational resources, which can be a barrier for massive labeling tasks. Exploring optimized inference or hybrid approaches combining LLMs with traditional methods is suggested.
- Downstream Applications: The evaluation focused on general reader preference. Future work could assess LLM label utility in specific downstream applications or through expert assessments.
Summary updated