TL;DR
Researchers evaluated how well modern LLMs can identify the rhetorical structure of 18th-century petitions in English and Swedish. While models like GPT-4 and LLaMA 3-70B excel at identifying formal greetings, pinpointing the actual "plea" (the Petitio) remains a significant challenge, particularly in Swedish and for smaller open-source models.
What Problem Does This Paper Address?
Historical documents like petitions are goldmines for historians, offering a rare glimpse into the lives of ordinary people in pre-democratic societies. However, these texts are often "messy"—featuring non-standard orthography, heavy abbreviations, and complex rhetorical flourishes.
To extract information at scale, scholars need to segment these documents into functional parts: the greeting (Salutatio), the request (Petitio), and the closing (Conclusio). Manually annotating thousands of these is labor-intensive. This paper investigates whether LLMs can automate this fine-grained, content-aware annotation task across different languages and model scales, and how well model "confidence" aligns with human-perceived difficulty.
Methods
The authors employed a few-shot prompting strategy, testing six prominent LLMs: GPT-4, Gemini 1.5 Pro, Mixtral 8x22B, Mistral 7B, and LLaMA 3 (8B and 70B).
The experimental design focused on three variables:
- Prompt Complexity: They compared "short 1-shot" (minimal instructions) vs. "long 1-shot" (detailed context) vs. "long 3-shot" (multiple examples).
- Cross-Linguistic Performance: They tested Swedish data using both Swedish and English instructions to see if the prompt language influenced the tagging of a moderately resourced language.
- Uncertainty Alignment: Models were asked to provide a difficulty score (0-2), which was then compared against human annotator difficulty ratings using Spearman’s correlation and Mean Error.
Datasets
The study introduced a dataset of 200 historical petitions (100 English, 100 Swedish) from the 18th century.
- English Set: Sourced from the London Lives 1690–1800 archive.
- Swedish Set: Sourced from the Gender and Work (GaW) project at Uppsala University.
- Gold Standard: Each document was manually annotated by two experts to mark the boundaries of the Salutatio, Petitio, and Conclusio. Human annotators also assigned a difficulty score based on how much "extended discussion" was needed to reach a consensus on the boundaries.
Key Insights and Ideas
- The "Salutatio" is a Low-Hanging Fruit: Across almost all models and languages, identifying the formal greeting was highly successful (F1 scores often in the high 90s). Its formulaic nature makes it an easy win for LLMs.
- The "Petitio" is the Frontier: Identifying the actual request is much harder because it is often linguistically "blended" with the Narratio (the story leading up to the request). Smaller models frequently failed to find the boundary where the story ends and the plea begins.
- Prompt Robustness: Surprisingly, increasing prompt complexity (more detailed instructions or more examples) did not systematically improve performance. This suggests that for this specific rhetorical task, LLMs are relatively robust to prompt design, or perhaps they hit a performance ceiling early.
- The "Dunning-Kruger" Effect in LLMs: Models consistently rated documents as "easier" than humans did (negative Mean Error). There was a very weak correlation between what a human found difficult (e.g., non-standard spelling or ambiguous phrasing) and what the model struggled with.
Key Results
- Top Performers: GPT-4 and LLaMA 3-70B were the most consistent across both languages. In English, they achieved F1 scores in the mid-to-high 90s for all sections.
- Language Gap: Performance was generally lower on Swedish petitions than English ones. For example, on the Petitio section, Mistral 7B’s F1 score dropped from ~79% in English to ~60% in Swedish.
- Model Scale Matters: There was a sharp performance drop-off when moving from 70B+ parameter models to 7B-8B models, particularly for the Petitio and Conclusio tags.
- Cross-Lingual Success: Using English prompts to annotate Swedish text was surprisingly effective, showing little systematic disadvantage compared to using Swedish prompts.
Limitations and Future Work
The authors acknowledge that the dataset is relatively small (200 documents), which limits the ability to perform extensive fine-tuning. Furthermore, the study only focused on three rhetorical categories, omitting the Exordium and Narratio because they are notoriously subjective even for humans to distinguish.
Future Directions: 1. Fine-tuning: Moving beyond prompting to domain-specific fine-tuning might help models better calibrate their difficulty ratings and handle the "blended" boundaries of the Petitio. 2. Downstream Tasks: Exploring how this automated segmentation improves subsequent tasks like Named Entity Recognition (NER) or social network reconstruction from historical archives. 3. Broader Language Support: Expanding the methodology to even lower-resourced languages to test the limits of cross-lingual transfer in digital humanities.
Summary updated