TL;DR
This paper explores using large language models (LLMs) like Llama 2 for correcting errors in Optical Character Recognition (OCR) of historical newspapers. They demonstrate that a prompt-based approach with an instruction-tuned Llama 2 significantly outperforms traditional sequence-to-sequence models like BART, achieving a substantial reduction in character error rates. This work highlights the potential of LLMs to unlock valuable historical archives for research.
What Problem Does This Paper Address?
Digitizing historical documents, especially newspapers, is crucial for research, but the resulting OCR text is often riddled with errors. These errors stem from the inherent challenges of old paper: ink bleed, fading, unusual fonts, and complex layouts. This "noisy" OCR text significantly hinders text mining, keyword searches, and overall accessibility for humanities scholars. While sequence-to-sequence models have been the standard for post-OCR correction, this paper investigates whether the advanced capabilities of modern generative LLMs can offer a more effective solution.
Methods
The authors compare two main approaches for post-OCR correction:
- Sequence-to-Sequence (Seq2Seq) Fine-tuning: They fine-tune BART, a well-established Transformer-based model, on a neural machine translation task. This is the prevalent method in the field, where the model learns to map erroneous OCR text to correct text.
- Prompt-Based LLM Adaptation: They leverage Llama 2, a powerful decoder-only LLM, and adapt it using an instruction-tuning approach. This involves creating a dataset with specific prompts (e.g., "Fix the OCR errors in the provided text") and training the LLM to generate the corrected output. They specifically use LoRA (Low-Rank Adaptation) to efficiently fine-tune Llama 2.
Datasets
- BLN600: This is the primary dataset used in the paper. It's a parallel corpus of 19th-century British newspaper articles, containing both OCR text and manually re-keyed human transcriptions. A key characteristic of BLN600 is that the OCR text and ground truth are not aligned at the character level, which presents a challenge for some models. The authors address this by splitting the ground truth into segments and finding corresponding OCR text.
- ICDAR Corpora: The authors mention that they initially considered using the ICDAR2017 and ICDAR2019 corpora for post-OCR correction. However, they decided against it due to concerns about potential data contamination, as excerpts from literary works within these datasets might be present in the training data of LLMs like Llama 2.
Key Insights and Ideas
- LLMs Excel at Prompt-Based Correction: The core insight is that generative LLMs, when instruction-tuned, can effectively perform post-OCR correction using a prompt-based approach, outperforming traditional Seq2Seq models.
- Limited Data Advantage for LLMs: Foundation models like Llama 2 show a significant advantage when working with limited training data. They exhibit strong performance from the outset, whereas Seq2Seq models like BART benefit more from larger training sets. This is attributed to the extensive pre-training of LLMs.
- Character Error Rate (CER) as a Key Metric: The paper consistently uses CER to measure performance, which is a standard metric for evaluating text recognition accuracy.
- Handling Unaligned Data: The BLN600 dataset's lack of character-level alignment between OCR and ground truth is a practical challenge. The authors' segmentation approach demonstrates a viable strategy for preparing such data for Seq2Seq models.
Key Results
- Llama 2 Significantly Outperforms BART: The top-performing model, Llama 2 13B, achieved a 54.51% reduction in character error rate (CER) on the test set.
- BART's Performance: The fine-tuned BART model (large variant) achieved a 23.30% reduction in CER.
- Performance vs. Training Data Size: Figure 3 illustrates that Llama 2's performance plateaus early with increasing training data, while BART shows a more gradual improvement, highlighting LLMs' efficiency with smaller datasets.
- Error Type Correction: Table 2 shows examples of corrections made by Llama 2 13B, indicating its ability to fix substitutions, deletions, and insertions, with substitutions and deletions being the easiest to correct.
Limitations and Future Work
- Language Specificity: The authors acknowledge that while Llama 2 is powerful, its primary training on English data might pose challenges for adapting it to post-OCR correction in other languages. They suggest that multilingual Seq2Seq models like mBART might be more suitable for such scenarios.
- Data Contamination Concerns: The decision to avoid ICDAR corpora due to potential data contamination in LLM training data is a significant consideration for future work.
- Future Directions:
- Building an assistant model using Llama 2's 'chat' version to explain error corrections, enhancing trustworthiness and human verification.
- Quantifying the model's confidence in its corrections to flag potentially problematic ones for review.
- Exploring the use of LLMs for post-OCR correction in multilingual contexts.
Summary updated