Unlocking 18th- and 19th-century Playbills with AI: An Experiment in Qualitative Data Categorization

Deven M. Parker, Kaiwen Zheng, Michael T. Gamer, Joemon M. JoseUmanistica 2025 · DOI

TL;DR

This paper explores the use of Large Language Models (LLMs) like ChatGPT-4o to automatically categorize and extract detailed information from historical British playbills from the 18th and 19th centuries. The authors demonstrate that LLMs can effectively handle the structured, yet complex, data within playbills, offering a scalable solution for digitizing and analyzing these historical documents, while also uncovering new insights into their intricacies.

What Problem Does This Paper Address?

The humanities are increasingly exploring the potential of AI, particularly LLMs, for tasks like classification and categorization of historical texts and archives. However, a key challenge is extracting detailed information from these documents without losing the inherent complexity and ambiguity that makes them valuable. This paper specifically addresses how LLMs can be used to categorize and tag qualitative data from historical playbills, which are rich in nested information (e.g., play titles, performers, roles, genres). The authors aim to replicate the meticulous process of manual tagging, preserving the intricate data structures of playbills, and to see if LLMs perform better on data that already has an implicit classification structure.

Methods

The core of the methodology involves using LLMs, specifically ChatGPT-4o, to automate the extraction and categorization of data from playbills. The authors developed a "tailored prompt engineering approach" to systematically transform visual textual information into structured JSON data. This approach consists of three main components:

  1. Initialization Setup: Defines the LLM's role (e.g., "expert researcher with deep expertise in theatrical playbills") and provides explicit task instructions for analyzing textual details and converting them into JSON.
  2. Output Formatting: Provides a detailed example of the desired JSON output structure, illustrating how each extracted element should be organized into nested components like ephemeralRecord, basicInfo, shows, performances, and ticketInformation.
  3. Semantic Guidance: Incorporates domain-specific instructions to guide the LLM in interpreting historically grounded data. This includes rules for metadata, contributor policies, attraction policies, and location limitations, ensuring accurate and contextually relevant extraction.

The authors also conducted a comparative analysis of three LLMs (Llama 3.2 Vision, Mistral-OCR, and ChatGPT-40) to determine the most suitable model for their task, ultimately selecting ChatGPT-40 for its robustness and reliability.

Datasets

The study utilized a dataset of 100 playbills from the Theatres Royal Bath, Birmingham, and Bristol (1795-1843) for initial testing, for which hand-tagged YAML records were already available. This allowed for direct comparison between manual and AI-generated tags. Subsequent testing involved more complex playbills with issues like missing years, damage, handwritten corrections, and detailed descriptions. The authors also mention locating and photographing over 500,000 playbills from archives across the UK and US, indicating the scale of their overall project.

Key Insights and Ideas

  • Playbills as Ideal Candidates for LLM Analysis: The inherent structured nature of playbills (date, theatre, play title, performers, roles) makes them particularly well-suited for LLM-based categorization and tagging, compared to more free-form texts.
  • LLMs Can Preserve Nested Data Structures: The prompt engineering approach successfully enabled LLMs to capture the complex, nested relationships within playbills, such as linking performers to their specific roles.
  • Uncovering Hidden Complexities: While LLMs can automate categorization, the process also revealed unexpected layers of complexity and ambiguity in the playbill data, especially concerning genre classification.
  • Genre Classification is a Major Challenge: The LLM struggled with "ourGenreAttribution" because it tended to accept the playbill's stated genre at face value, rather than applying an expert's interpretive classification, which is crucial given the historical practice of marketing plays under different genres.
  • Image Quality is Paramount: The performance of LLMs is heavily dependent on the quality of the input images. Damaged, blurry, or low-resolution playbills significantly hinder accurate OCR and data extraction.

Key Results

  • High Accuracy for Straightforward Data: LLMs achieved high accuracy (100%) for straightforward categories like "Date" and "Theatre location."
  • Good Performance on Complex Data: Categories like "Play title" (92.92%) and "Printer information" (88%) also showed strong performance.
  • Challenges with Ambiguous Data: "Theatre name" (52% correct) and "Our genre attribution" (44.17% correct) were the weakest categories, highlighting the LLM's difficulty with interpretation and nuanced classification.
  • Time Savings: The automated approach offers significant time savings compared to manual tagging, which can take 20-45 minutes per playbill. This is crucial for large-scale projects involving hundreds of thousands of records.

Limitations and Future Work

  • Image Quality: The primary limitation is the reliance on image quality. Blurry, damaged, or low-resolution playbills significantly impact extraction accuracy.
  • Handwritten Modifications: LLMs struggle to distinguish between original text and handwritten corrections, leading to potential misinterpretations.
  • Ambiguities and Missing Information: Inconsistent or missing information within playbills (e.g., missing year) poses challenges for LLMs that lack external context.
  • Genre Ambiguity: The LLM's inability to perform nuanced genre attribution, as discussed, requires further human intervention.
  • "Black Box" Nature of LLMs: The proprietary nature of models like ChatGPT-40 limits transparency regarding training data, potential biases, and reproducibility.
  • Future Work: The authors suggest ongoing refinement of prompt engineering, regular expert review and manual validation, and further research into how transformer architectures can better account for the inherent instability and ambiguity of humanistic data. They also emphasize the need for robust image enhancement and human verification for challenging sources.

Summary updated

Abstract

Recent advances in large language models (LLMs) have generated excitement from scholars in the humanities about the ways our research might be transformed with their successful integration, particularly for tasks that require multiclass classification or categorization. This paper offers an interdisciplinary exploration by researchers in the fields of drama and computer science to use generative AI techniques to categorize, tag, and analyze qualitative information from eighteenth- and nineteenth-century British playbills. Our goal was to replicate the process of manually tagging playbills via categorization in a manner that preserves playbills’ intricate nested data structures. Our results indicate that there is promising work to be done on humanities data where there is already an underlying classification structure in place; at the same time, our attempts revealed unexpected layers of complexity and ambiguity in this dataset, particularly around genre categorization and nested performance data. Ultimately, we wish to highlight not just LLMs’ capacity to categorize historical data at scale but also to shed new light on that data’s existing intricacies.
BibTeX
@inproceedings{parker2025unlocking,
  author = {Deven M. Parker and Kaiwen Zheng and Michael T. Gamer and Joemon M. Jose},
  title = {{Unlocking 18th- and 19th-century Playbills with AI: An Experiment in Qualitative Data Categorization}},
  booktitle = {Umanistica},
  year = {2025},
  doi = {10.60923/issn.2532-8816/21718},
}
Figure