LLM_annotate: A Python package for annotating and analyzing fiction characters

Hannes Rosenbusch

Abstract

LLM_annotate is a Python package for analyzing the personality of fiction characters with large language models. It standardizes workflows for annotating character behaviors in full texts (e.g., books and movie scripts), inferring character traits, and validating annotation/inference quality via a human-in-the-loop GUI. The package includes functions for text chunking, LLM-based annotation, character name disambiguation, quality scoring, and computation of character-level statistics and embeddings. Researchers can use any LLM, commercial, open-source, or custom, within LLM_annotate. Through tutorial examples using The Simpsons Movie and the novel Pride and Prejudice, I demonstrate the usage of the package for efficient and reproducible character analyses.

Introduction

Analyzing the behaviors and personalities of characters in literary works or scripts is a cornerstone of humanities and social science research. Traditionally, scholars faced a difficult choice: invest thousands of hours in manual human annotation or utilize "bag-of-words" computational methods. While manual coding is prohibitively expensive, keyword-based tools often fail to capture the semantic nuance of characterization—for instance, missing a character's "authoritarianism" if the text describes a stern look rather than using the specific word "authority."

Large Language Models (LLMs) offer a middle ground, capable of deep semantic reading at scale. However, the lack of a standardized, transparent workflow has hindered their adoption in formal character analysis. LLM_annotate addresses this gap by providing a reproducible Python-based pipeline for character study.

The LLM_annotate Framework

The package implements a modular five-step pipeline designed to move from raw text to validated character statistics.

The LLM_annotate pipeline architecture, showing the flow from raw text to JSON outputs and a human-in-the-loop GUI.

1. Text Chunking

Large texts are processed using the chunk_text function, which breaks documents into manageable segments (defaulting to 500 tokens). To preserve narrative flow and context, the function prepends the final three sentences of the previous chunk to the current one, ensuring the LLM maintains a "memory" of immediate preceding events.

2. Character Annotation

The annotate function sends these chunks to an LLM (GPT-4o by default, though the package is LLM-agnostic). The model identifies character actions and infers traits. This can be performed as an exploratory task or a targeted study where researchers provide specific trait definitions and examples to guide the model.

3. Character Disambiguation

A common challenge in automated analysis is that LLMs may treat "Homer" and "Homer Simpson" as distinct entities. The disambiguate function uses the LLM to identify potential pseudonyms and merge character profiles.

Example of AI-generated pseudonym lists used for character disambiguation in The Simpsons Movie.

As shown in the figure above, the system identifies overlaps such as "Wiggum" and "Police chief." Crucially, the package allows researchers to manually override these merges to ensure accuracy.

4. Human-in-the-Loop Quality Assessment

To maintain scientific rigor, LLM_annotate includes a built-in GUI for quality control. The score_annotations function presents a random sample of annotations to a human reviewer, who rates them as Correct, Questionable, or Incorrect. This allows for the calculation of inter-rater agreement and validates the model's performance before final analysis.

5. Statistical Analysis

Finally, the compute_annotation_statistics function generates character-level statistics, trait scores, and high-dimensional embeddings, facilitating downstream quantitative research.

Case Studies and Results

The package was validated using two distinct datasets: The Simpsons Movie script and Jane Austen’s Pride and Prejudice.

  • Accuracy: In the Simpsons Movie trial, a human reviewer validated the LLM's output, rating 95% of the generated annotations as "completely correct."
  • Psychological Mapping: In the Pride and Prejudice study, the tool successfully mapped characters along social dimensions like "Social Dominance Orientation" (SDO) and "Agreeableness." For example, Lady Catherine de Bourgh was correctly identified as high in SDO and low in Agreeableness, while Elizabeth and Jane Bennet clustered at the opposite end of the spectrum.
  • Sensitivity Analysis: Comparing 500-token chunks against chapter-wise chunking revealed that while results were statistically similar, the smaller default chunks produced significantly more data points (1,981 vs. 578 annotations), leading to higher precision in character statistics.

Discussion and Limitations

A primary strength of LLM_annotate is its LLM-agnosticism, allowing researchers to plug in any model (OpenAI, Anthropic, or local open-source models). However, the author notes that LLMs are not a "set it and forget it" solution. Human oversight remains essential, particularly for identifying omissions (what the LLM missed) and navigating subtle cultural or historical nuances that a model might misinterpret.

Future work will focus on extending the package to handle multi-modal inputs and integrating more complex sensitivity analysis tools directly into the workflow to further formalize interpretive rigor in digital humanities.

Summary updated

Abstract

LLM_annotate is a Python package for analyzing the personality of fiction characters with large language models. It standardizes workflows for annotating character behaviors in full texts (e.g., books and movie scripts), inferring character traits, and validating annotation/inference quality via a human-in-the-loop GUI. The package includes functions for text chunking, LLM-based annotation, character name disambiguation, quality scoring, and computation of character-level statistics and embeddings. Researchers can use any LLM, commercial, open-source, or custom, within LLM_annotate. Through tutorial examples using The Simpsons Movie and the novel Pride and Prejudice, I demonstrate the usage of the package for efficient and reproducible character analyses.
BibTeX
@misc{rosenbusch2025llm,
  author = {Hannes Rosenbusch},
  title = {{LLM\_annotate: A Python package for annotating and analyzing fiction characters}},
  year = {2025},
  eprint = {2601.03274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03274},
}
Figure