On-the-fly Historical Handwritten Text Annotation

Ekta Vats, Anders Hast

Abstract

Training robust handwriting recognition algorithms for historical manuscripts is hindered by the labor-intensive nature of manual annotation and the poor quality of degraded documents. This paper introduces a semi-automated, "on-the-fly" annotation workflow that transforms rough user drag-and-drop selections into precise, noise-free bounding boxes. By leveraging box expansion, Gaussian filtering, and a connected-component-based heuristic, the system effectively filters out background noise and overlapping strokes. Experimental results on the Esposalles and Bentham datasets demonstrate significant improvements in annotation precision and speed, with relative corrections reaching over 50% in challenging cases.

Introduction

The development of information retrieval and handwriting recognition systems for historical archives requires vast amounts of accurately labeled "ground truth" data. However, historical manuscripts present significant challenges: ink bleed-through, fading, paper stains, and non-linear layouts often render standard OCR and annotation tools ineffective.

Example of a historical manuscript with complex layout and multiple entity types requiring annotation.

Existing platforms like GEDI or TrueViz, while useful for modern text, often struggle with the "messy" nature of historical scripts. Manual annotation in these environments is tedious and frequently results in imprecise bounding boxes that include noise from adjacent lines or background artifacts. This paper proposes a method to bridge the gap between the need for high-quality labels and the practical constraints of manual labor.

Methodology

The authors propose a semi-automated workflow designed to prioritize both speed and precision. The process begins with a "lazy" user selection and ends with a refined word image.

The proposed semi-automated workflow for refining user annotations into cleaned bounding boxes.
  1. User Input: A user performs a rough drag-and-drop selection over a target word. This approach allows for "lazy" selections, reducing the cognitive load on the annotator.
  2. Box Expansion: To ensure all strokes (such as descenders in 'g' or 'p') are captured, the system automatically expands the initial selection by 1/3 in width and 2/3 in height. This is based on the observation that word height is relatively constant within a line.
  3. Preprocessing: Gaussian filtering is employed to normalize the image snippet and mitigate background noise.
  4. Connected Components (CC) Extraction: The system identifies distinct "ink blobs" within the expanded region.
  5. The 1% Heuristic: To distinguish the target word from surrounding noise or bleed-through, each component is evaluated. If a component's area exceeds 1% of the total rectangle area, it is retained as part of the word. This simple heuristic is computationally efficient compared to deep learning models.
  6. Refinement: The final bounding box is shrunk to encapsulate only the validated components, producing a "noise-free" output.

Experimental Evaluation

The method was benchmarked against two significant historical datasets:

  • Esposalles Dataset (BH2M): 17th-century marriage records from Barcelona Cathedral, characterized by a single author's consistent style in old Catalan.
  • KWS-2015 Bentham Dataset: Messy 18th and 19th-century manuscripts by Jeremy Bentham, featuring diverse styles, varying font sizes, and frequent strike-throughs.

Results and Discussion

The system proved highly effective at "tightening" human annotations. The primary metric used was Relative Correction, representing the percentage of noise removed from the initial human selection.

  • Precision Gains: In the Esposalles dataset, relative correction reached up to 30.95%, indicating that nearly a third of the manual selection was unnecessary noise.
  • Handling Complexity: On the more challenging Bentham dataset, the system achieved relative corrections as high as 53.96%, effectively removing over half of the area from the initial rough selection.
  • Efficiency: By allowing for imprecise manual input, the tool significantly accelerates the annotation pipeline without compromising the quality of the resulting training data.

Conclusion

This "on-the-fly" annotation method provides a practical solution for generating high-quality ground truth from degraded historical documents. While currently limited by a reliance on global thresholding—which can occasionally clip faint strokes—future work will focus on adaptive thresholding and parameter optimization. Expanding this logic to character-level annotation remains a key objective for future iterations.

Summary updated

Abstract

The performance of information retrieval algorithms depends upon the availability of ground truth labels annotated by experts. This is an important prerequisite, and difficulties arise when the annotated ground truth labels are incorrect or incomplete due to high levels of degradation. To address this problem, this paper presents a simple method to perform on-the-fly annotation of degraded historical handwritten text in ancient manuscripts. The proposed method aims at quick generation of ground truth and correction of inaccurate annotations such that the bounding box perfectly encapsulates the word, and contains no added noise from the background or surroundings. This method will potentially be of help to historians and researchers in generating and correcting word labels in a document dynamically. The effectiveness of the annotation method is empirically evaluated on an archival manuscript collection from well-known publicly available datasets.
BibTeX
@inproceedings{vats2017fly,
  author = {Ekta Vats and Anders Hast},
  title = {{On-the-fly Historical Handwritten Text Annotation}},
  booktitle = {HDI@ICDAR},
  year = {2017},
  doi = {10.1109/icdar.2017.374},
}
Figure