The Paleographer's Eye ex machina: Using Computer Vision To Assist Humanists in Scribal Hand Identification

Samuel Grieggs, C. E. M. Henderson, Sebastian Sobecki, Alexandra Gillespie, Walter ScheirerWACV 2024

TL;DR

This paper introduces a user-friendly computer vision tool designed to assist paleographers in identifying scribal hands in medieval manuscripts. By providing visual feedback on script similarity, the software aims to bridge the gap between traditional, qualitative paleography and the quantitative power of machine learning, making advanced analysis accessible to humanities scholars.

What Problem Does This Paper Address?

Paleography, the study of historical handwriting, traditionally relies on the subjective "eye" of experts to attribute manuscripts to specific scribes, periods, or regions. While the digitization of manuscripts has opened new avenues for research, the lack of a standardized methodology and the inherent subjectivity of human analysis pose significant challenges. Existing computer vision approaches for writer identification often produce abstract numerical scores that are difficult for paleographers to interpret or integrate into their qualitative assessments. This paper addresses the need for a tool that translates complex computational analysis into visually understandable insights, empowering paleographers without requiring them to become machine learning experts.

Methods

The core of the proposed solution is a software package that uses computer vision to compare image patches of a queried document against a pre-built template of a known scribe's handwriting. The system employs a ResNet-50 model, initialized with ImageNet weights and fine-tuned on historical script datasets. For analysis, the software breaks down images into 64x64 pixel patches. It then uses a template-matching approach, comparing features extracted from these patches to a template derived from a known scribe's samples. The output is a visual overlay on the document, where green patches indicate high similarity, yellow moderate similarity, and red low similarity to the template. This allows paleographers to quickly identify regions that are most indicative of a particular scribe's hand. The software also features a user-friendly GUI and is distributed as a standalone executable for ease of use across different operating systems.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Privy Seal Dataset: A newly collected dataset of 14 scribes from the English central government's Privy Seal Office, featuring highly similar scripts in Latin, French, and English. This dataset contains approximately 146 images.
  • IAM Handwriting Database: A standard dataset of modern English handwritten text, used for training and as a portion of the training dataset.
  • CVL Database: A dataset of 1604 page images from 310 writers, used for evaluating the tool on modern text.
  • 2017 ICDAR Handwriting Identification Challenge Data: An additional dataset used for evaluation, comprising 4782 images from 1114 writers.
  • Additional Training Images: Images sourced from the "Subscriptions to the oath" in the Common Paper of the Company of Scriveners of the City of London, and other publicly available sources, providing ground truth for 164 Scriveners and 24 other identities.

Key Insights and Ideas

  • Bridging the Human-Computer Divide: The primary insight is that advanced computer vision can be made accessible and useful to humanities scholars by focusing on visual interpretability rather than just raw accuracy scores.
  • Augmenting, Not Automating: The tool is designed to "augment the paleographer's eye," providing an additional feature for analysis rather than replacing human expertise.
  • Visualizing Similarity: The color-coded overlay is a crucial innovation, translating complex similarity metrics into an intuitive visual representation that paleographers can readily understand.
  • Accessibility is Key: Distributing the software as a standalone executable with minimal dependencies significantly lowers the barrier to entry for non-technical users.
  • Context Matters: The Privy Seal dataset, focusing on scribes within a specific institutional context and writing similar scripts, presents a realistic challenge for distinguishing individual hands.

Key Results

The paper demonstrates the tool's utility through a case study involving the poet and clerk Thomas Hoccleve.

  • Qualitative Success: The tool successfully distinguished between Hoccleve's hand and that of other scribes, aligning with expert paleographic opinions in cases of disagreement (e.g., correctly identifying a manuscript as unlikely to be Hoccleve's).
  • Performance on Datasets: While the tool's performance on the CVL dataset did not reach state-of-the-art, it was considered a sanity check. The trained models showed promising results on the Privy Seal dataset, particularly when analyzing individual patches.
  • Distinguishing Similar Scripts: The case study highlighted the tool's ability to differentiate between scribes writing very similar scripts, a key challenge in paleography.

Limitations and Future Work

  • Limitations: The authors acknowledge that the performance on benchmark datasets like CVL is not state-of-the-art, as the primary goal was usability and interpretability for paleographers. They also note that a high similarity score alone might not be convincing to paleographers without further explanation of the underlying paleographic features.
  • Future Work: The authors plan to continue developing the software as an open-source project. Future work includes incorporating a wider variety of algorithms (e.g., siamese networks, transformer-based models), adding more sophisticated layout analysis tools for better text alignment, and developing methods to explicitly measure paleographic features (like letter angles) to provide more detailed explanations for the model's attributions.

Summary updated

BibTeX
@inproceedings{grieggs2024paleographer,
  author = {Samuel Grieggs and C. E. M. Henderson and Sebastian Sobecki and Alexandra Gillespie and Walter Scheirer},
  title = {{The Paleographer's Eye ex machina: Using Computer Vision To Assist Humanists in Scribal Hand Identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure