\textitTexT TexT - Text Extractor Tool for Handwritten Document Transcription and Annotation

Anders Hast, Per Cullhed, Ekta Vats

TL;DR

This paper introduces TexT, a user-friendly tool for semi-automatic transcription of historical handwritten documents. By combining interactive user input with a word spotting system, TexT significantly speeds up the transcription process and enables on-the-fly annotation, making large historical archives more accessible.

What Problem Does This Paper Address?

The digitization of historical documents is crucial for making vast amounts of historical information accessible for research and public consumption. While Optical Character Recognition (OCR) has revolutionized the handling of printed texts, handwritten documents remain a significant challenge. Manual transcription is labor-intensive, expensive, and prone to human error. Existing Handwritten Text Recognition (HTR) technologies, while improving, often struggle with the variability and non-uniformity of historical handwriting, especially for large-scale projects. This leaves a substantial portion of our historical record locked away in unsearchable formats. The paper addresses this gap by proposing a more efficient and user-friendly approach to transcribe and annotate these valuable historical texts.

Methods

TexT employs a semi-automatic approach that leverages a word spotting system for efficient transcription. The core idea is that users transcribe each unique word only once. The system then uses this input to find all other occurrences of that same word within the document on-the-fly. This significantly reduces the repetitive effort of manual transcription.

Key methodological contributions include:

  • Interactive Word Spotting: The system actively searches for and highlights other instances of a word as the user types it, providing immediate feedback.
  • On-the-fly Annotation: TexT allows users to annotate transcribed words with relevant metadata (e.g., names, places) and automatically generates ground truth labels.
  • Dynamic Bounding Box Adjustment: User-annotated bounding boxes are dynamically adjusted to perfectly encapsulate the transcribed word.
  • Background Noise Removal: A two-band pass filtering approach is used to remove background noise from word images, improving the clarity of the transcribed text.
  • Connected Components Extraction: This is performed after noise removal to isolate the word for further processing.

The approach builds upon the authors' previous work on on-the-fly annotation and integrates a segmentation-free word spotting algorithm.

Datasets

The proposed framework is tested on the Esposalles dataset, which is a subset of the Barcelona Historical Handwritten Marriages (BH2M) database.

  • BH2M Database: Contains information on 550,000 marriages registered between the 15th and 19th centuries.
  • Esposalles Dataset: Specifically consists of historical handwritten marriage records from the Barcelona cathedral, written in old Catalan between 1617 and 1619 by a single writer. The dataset comprises 174 pages, with 50 pages selected from the 17th century for this study.

The authors also mention plans to use ancient manuscripts from the Uppsala University Library for future experimentation.

Key Insights and Ideas

  • Human-AI Collaboration is Key: The paper strongly advocates for the combination of crowdsourcing (human input) and HTR (AI) as a crucial factor for tackling large-scale historical document transcription. This synergy is seen as the most effective way to bridge the gap between IT technologies and the need to preserve cultural heritage.
  • "Transcribe Once, Find Everywhere": The core innovation of TexT lies in its ability to drastically reduce transcription effort by identifying and presenting all instances of a word after it's transcribed just once. This "word spotting" approach is a significant departure from line-by-line or word-by-word manual transcription.
  • Interactive Learning for HTR: The system learns from user feedback in real-time. By confirming correct transcriptions or correcting errors, users actively improve the HTR engine's performance for that specific document and handwriting style.
  • Focus on User Experience: The emphasis on a "simple yet informative, and user-friendly GUI" is a deliberate design choice to encourage participation and make the often tedious task of transcription more engaging.

Key Results

The paper demonstrates the effectiveness of TexT through experimental results, though specific quantitative metrics like transcription accuracy percentages are not explicitly detailed in the provided text. However, the qualitative results and the system's functionality are highlighted:

  • Efficient Transcription: The system is designed for "quick document transcription."
  • On-the-fly Word Finding: Figures 2 and 3 visually demonstrate the system's ability to find multiple occurrences of a marked word automatically.
  • Cleaned Transcribed Words: Figure 1b shows a transcribed word with background noise effectively removed, making it easier to read.
  • Annotation Capabilities: Figure 4 illustrates ongoing transcription with users annotating names and places using different colors, showcasing the tool's annotation features.
  • Speed: The authors mention that searching one word in their MATLAB implementation takes about 2 seconds for the example shown in Figure 2.

Limitations and Future Work

Limitations:

  • HTR Performance on New Documents: The paper acknowledges that deep learning techniques for HTR typically require several hundred annotated examples, which are not usually available when starting the transcription of an entirely new document. This implies that TexT's initial performance on a completely new manuscript might rely more heavily on the user's manual input and the word spotting algorithm.
  • Line Detection Challenges: The performance of some related HTR methods is dependent on accurate text line detection, which is noted as a challenging task for historical handwritten documents. While TexT's word spotting approach might mitigate this to some extent, it's a general challenge in the field.

Future Work:

  • Client-Server/Cloud-Based Solution: To further reduce latency, the authors intend to implement a transcription framework that can operate on a client-server or cloud-based architecture.
  • Implementation on Ancient Manuscripts: The plan is to implement the transcription framework on ancient manuscripts from Uppsala University Library, which will likely present new challenges and opportunities.
  • Parallel Processing for Large Documents: The system will be optimized to split the search work in parallel on a dedicated computer, enabling the search of even large documents in a matter of seconds.
  • Exploring Different Learning Methods: The authors are evaluating different learning methods to improve the transcription algorithm, with a particular interest in deep learning when sufficient annotated data becomes available.
  • Ranking and Merit Systems: The project planning includes a ranking system combined with a merit report for contributors, aiming to incentivize participation.
  • Proof-reading and Ingestion: A robust proof-reading structure and a safe ingestion mechanism for the digital repository are planned.

Summary updated

Abstract

This paper presents a framework for semi-automatic transcription of large-scale historical handwritten documents and proposes a simple user-friendly text extractor tool, \(\textit{TexT}\) for transcription. The proposed approach provides a quick and easy transcription of text using computer assisted interactive technique. The algorithm finds multiple occurrences of the marked text on-the-fly using a word spotting system. \(\textit{TexT}\) is also capable of performing on-the-fly annotation of handwritten text with automatic generation of ground truth labels, and dynamic adjustment and correction of user generated bounding box annotations with the word being perfectly encapsulated. The user can view the document and the found words in the original form or with background noise removed for easier visualization of transcription results. The effectiveness of \(\textit{TexT}\) is demonstrated on an archival manuscript collection from well-known publicly available dataset.
BibTeX
@inproceedings{hast2018textittext,
  author = {Anders Hast and Per Cullhed and Ekta Vats},
  title = {{\textbackslash\{\}textitTexT TexT - Text Extractor Tool for Handwritten Document Transcription and Annotation}},
  booktitle = {IRCDL},
  year = {2018},
  url = {https://openreview.net/forum?id=NHDy9UyFUd},
}
Figure