Abstract
The digitization of historical archives presents a significant challenge for automated text recognition due to the extreme diversity and non-stationarity of handwritten scripts. This paper describes the Monk system, a lifelong learning e-Science service developed at the University of Groningen. Monk utilizes a tiered machine learning strategy, termed the "Ball-park principle," which transitions from rule-based attribute learning to deep neural networks as human-verified labels accumulate. By integrating a human-in-the-loop "Fahrkunst" process, the system enables efficient indexing of massive, messy datasets. We report on the system's architecture, the scale of its current operations, and key performance metrics across diverse historical collections.
The Challenge of Historical Manuscripts
While Optical Character Recognition (OCR) for modern printed text is a mature technology, Handwritten Text Recognition (HTR) for historical manuscripts remains a significant hurdle. Standard AI benchmarks, such as MNIST or the IAM dataset, often fail to predict performance on real-world historical data like 17th-century Russian pamphlets or Egyptian papyri.
Historical archives present four primary challenges, often referred to as the "Four Vs" of Big Data: Volume (massive numbers of pages), Velocity (rapid influx of new scans), Variety (diverse scripts and layouts), and Veracity (uncertain ground truth). Furthermore, handwriting is "non-stationary"; a single scribe’s style can drift significantly over time, becoming more abbreviated or sloppier as a journal progresses.
Methodology: The Monk System Architecture
To address these challenges, the Monk system employs a modular, lifelong learning architecture rather than a static model. Two core principles guide its operation:
The Ball-park Principle
This tiered approach selects the optimal machine learning model based on the density of available training labels:
- Zero labels: Rule-based attribute learning detects specific radicals or strokes for "zero-shot" recognition.
- 1–5 labels: Employs Nearest-Neighbor (1NN) or Bayesian modeling.
- 20–100 labels: Transitions to Support Vector Machines (SVMs).
- 100+ labels: Deploys deep learning (CNNs) once data density justifies the model complexity.
The "Fahrkunst" Principle
Named after the reciprocating ladders used in old mines, this principle describes an iterative human-in-the-loop process. The system generates ranked "hit lists" for users to confirm. These confirmations are fed back into the system, triggering model re-computation within 24 hours. This creates a "snowball effect" where increased accuracy allows users to verify data much faster than manual transcription, leading to an avalanche of new training data.
Scale and Data Diversity
As of July 2018, the Monk system has scaled to accommodate a vast array of institutional collections. The system's strength lies in its ability to handle messy, mixed-density datasets rather than relying on small, specialized models.
Key Results and Insights
Field tests within the Monk environment have yielded several critical insights into large-scale HTR:
- Model Performance: In scenarios with high class counts (>1000) but low samples per class, traditional methods like Bag of Visual Words (BOVW) achieved 87% accuracy, outperforming CNNs at 83%. However, with sufficient label harvesting, deep end-to-end training achieved 95% word accuracy in specific books.
- Efficiency: The hit-list confirmation method allowed a single user to add approximately 30 labels per minute, a significant improvement over traditional word-by-word transcription.
- Metrics: The paper proposes the Equal Uncertainty Rate (EUR) as a more effective metric than the standard Equal Error Rate (EER) for open-set data mining, where the labels of non-target samples are often unknown.
Conclusion and Future Work
The Monk system demonstrates that a lifelong learning approach is essential for indexing the "long tail" of historical manuscripts. Future work will focus on incorporating linguistic and semantic resources to assist in recognizing rare words (hapax legomena) and improving interoperability between major platforms like Transkribus and the READ project to share transcription data and cross-verify results.
Summary updated