TL;DR
This paper explores how well Large Language Models (LLMs) can identify named entities (like people, places, and organizations) in historical texts using zero-shot learning. While LLMs show promise, their performance on historical texts is currently lackluster due to challenges like language evolution, OCR errors, and a tendency to "hallucinate" information not present in the source.
What Problem Does This Paper Address?
Named Entity Recognition (NER) is crucial for understanding text, but applying it to historical documents is particularly challenging. These texts often suffer from:
- Language Change: Vocabulary, spelling, and grammar evolve over time, making modern NER models struggle.
- Data Scarcity: Manually annotating historical texts is expensive and time-consuming, limiting the availability of labeled datasets for traditional supervised learning.
- OCR/HTR Errors: Optical Character Recognition and Handwritten Text Recognition introduce noise and inaccuracies.
- Domain Transfer: Models trained on modern data often perform poorly when applied to older texts with different contexts and naming conventions.
This paper investigates whether prompt-answering LLMs, which can perform tasks without explicit fine-tuning (zero-shot learning), can overcome these hurdles for historical NER.
Methods
The researchers employed a three-step methodology:
- Model Application: They ran several prompt-answering LLMs (including T0, T5, mT5, BLOOMZ, and Aya) on a historical newspaper dataset using zero-shot prompting.
- Error Analysis: A manual examination of the models' outputs was conducted to identify systematic error types.
- Mitigation and Re-evaluation: They attempted to address common errors, particularly hallucinations, by modifying prompts and then re-evaluating model performance.
A key aspect was adapting prompt templates to guide the LLMs in extracting specific entity types from historical text. They also explored the impact of model size by testing both smaller and larger versions of the same model families.
Datasets
The primary dataset used was the HIPE2020 dataset, which consists of historical newspaper texts from the 18th to 20th centuries in English, French, and German. These texts were manually annotated by human experts. The study focused on a coarse-grained tag set including:
- Persons (PERS)
- Time (TIME)
- Locations (LOC)
- Products (PROD)
- Organizations (ORG)
The dataset was split by language and time period, with specific token counts provided for each.
Key Insights and Ideas
- LLMs struggle with historical nuances: Despite their general capabilities, LLMs exhibit significant weaknesses when applied to historical NER in a zero-shot setting.
- Hallucinations are a major issue: A substantial portion of the entities extracted by the models were not present in the original text. This "hallucination" phenomenon significantly impacts performance.
- Prompt engineering offers marginal gains: Modifying prompts to explicitly guide the model to stick to the input text led to small improvements, but did not fundamentally solve the performance issues.
- Model size matters, but not always decisively: Larger model versions generally showed slight improvements, but the core challenges remained across different sizes.
- Over-generation is common: Many models tend to produce output even when it's incorrect or nonsensical, rather than providing no answer.
Key Results
- Lackluster overall performance: The LLMs tested did not achieve satisfactory results for reliable automatic annotation of archival text.
- Hallucination rates: For the TO++ model, about half of the extracted entities were hallucinations. Aya 101 performed better, with around 11% hallucinated entities.
- Impact of filtering hallucinations: Filtering out hallucinated entities led to a modest increase in the overall F1 score (around 5% for TO++ and slightly less for Aya 101).
- Prompt modification effects: Modifying prompts slightly improved the percentage of correct entities for TO++ and reduced hallucinations for Aya 101, but the overall performance remained low.
- Specific model issues: Some models (e.g., BLOOMZ) exhibited peculiar behavior, segmenting the entire prompt and input as output, making reliable evaluation impossible.
Limitations and Future Work
The authors acknowledge several limitations:
- Low F1 scores: The performance of many models was so poor (F1 < 1%) that they were excluded from further analysis.
- Difficulty in evaluation: Accurately evaluating NER spans in historical texts is inherently challenging due to variations and errors.
- Model tendency to over-generate: The consistent production of non-empty outputs, even when incorrect, hinders accurate assessment.
Future work suggested includes:
- Prompt engineering for few-shot NER: Investigating how prompt-based learning can be leveraged with a small number of examples for historical NER.
- Two-step prompting: Exploring a process where the prompt is provided first, followed by the text to be analyzed, to mitigate over-generation.
- Minimum data requirements: Determining the minimal data needed for effective few-shot or zero-shot learning in historical NER.
- Model family scaling: Experimenting with how different parameter sizes within model families affect performance.
- Training data enrichment: Ensuring LLMs have seen sufficient historical text and NER examples during their training.
Summary updated