Combining Human and Machine Transcriptions on the Zooniverse Platform

Daniel Hanson, Andrea SimenstadEMNLP 2018

TL;DR

This paper tackles the challenge of transcribing historical handwritten documents, which is crucial for archival research but slow and error-prone. They propose a hybrid approach that combines human transcribers with a pre-trained neural network model, aiming to improve both the speed and accuracy of the transcription process.

What Problem Does This Paper Address?

Digitizing historical archives often involves transcribing handwritten documents, a process vital for making them searchable and accessible. Traditional Optical Character Recognition (OCR) struggles with the inherent variability and noise in handwriting. While crowdsourcing platforms like Zooniverse have democratized transcription, they require a significant number of human volunteers per document to achieve reliable accuracy, making the process time-consuming. Modern machine learning models show promise but often lack generalization to new, unseen datasets. This paper addresses the need for a more efficient and accurate system by exploring how to effectively combine the strengths of human transcribers and machine learning models.

Methods

The core of the paper's contribution lies in developing and integrating a machine learning model into a human-in-the-loop transcription workflow.

  • Deep Neural Network (DNN) Model: The authors trained a deep neural network for handwritten text recognition.
  • Pre-training: The model was initially pre-trained on two publicly available datasets: the IAM Handwriting Database and the Bentham Collection. This provides a strong foundational understanding of handwriting patterns.
  • Pseudo-Online Re-training: The pre-trained model was then further trained on data from the "Anti-Slavery Manuscripts at the Boston Public Library" crowdsourcing project. Crucially, this re-training was done in a "pseudo-online" fashion, meaning the model processed new data in small batches, in the same order it was collected, mimicking a real-time learning scenario.
  • Predictive Testing: To evaluate the model's predictive accuracy, they predicted each new line of text before training the model on that specific batch of data.
  • Human-Machine Integration: The ultimate goal is to integrate this model into the Zooniverse platform. The proposed workflow involves presenting the machine's predicted transcription to human volunteers. Volunteers can then either confirm the prediction (if correct), correct it, or re-transcribe the line entirely.

Datasets

  • IAM Handwriting Database: Used for initial pre-training of the neural network.
  • Bentham Collection (University College, London): Also used for initial pre-training.
  • "Anti-Slavery Manuscripts at the Boston Public Library" (Zooniverse Project): This dataset was used for re-training the pre-trained model in a pseudo-online fashion.

Key Insights and Ideas

  • Hybridization is Key: The paper strongly advocates for combining human and machine efforts, recognizing that neither is perfect alone. Humans excel at accuracy and understanding context, while machines offer speed and scalability.
  • Pseudo-Online Training for Real-World Adaptation: Training the model on data in the order it was collected, rather than in a shuffled manner, is a novel approach for adapting models to evolving data streams, which is common in crowdsourcing.
  • Efficiency Gains through Prediction: Presenting predictions to volunteers can significantly speed up the process. If the machine is correct, the human simply confirms, saving considerable typing time.
  • Focusing Human Effort: This hybrid approach allows humans to focus their efforts on the most challenging or ambiguous transcriptions, where machine predictions are likely to be less accurate.
  • Continuous Improvement Loop: The proposed system creates a feedback loop where human corrections can be used to further refine the machine model, leading to ongoing improvements in accuracy.

Key Results

  • After training on 90,000 lines of text, the model achieved an error rate of 12% on previously unseen data.
  • While this error rate is higher than some studies using cleaner datasets, it's noted that the model performed identically to human performance in many cases, suggesting its utility for improving transcription speed.
  • The proposed integration aims to improve efficiency by allowing volunteers to quickly confirm correct predictions, thereby reducing the time spent on each line.

Limitations and Future Work

  • Error Rate vs. Human Experts: The 12% error rate is higher than the 2.5% error rate achieved by volunteers when compared to experts. The authors acknowledge this and suggest that the noisier, less curated nature of the training data might contribute to this difference.
  • Speed vs. Accuracy Trade-off: While the model can improve speed, it doesn't necessarily guarantee higher accuracy on its own compared to purely human efforts. The benefit comes from the combination.
  • Future Work: The paper explicitly plans to incorporate this model into the Zooniverse human transcription process. This integration is expected to leverage existing Zooniverse infrastructure and provide opportunities for efficiency gains. The authors also highlight that this process will allow for more focused model training on difficult text.

Summary updated

BibTeX
@inproceedings{hanson2018combining,
  author = {Daniel Hanson and Andrea Simenstad},
  title = {{Combining Human and Machine Transcriptions on the Zooniverse Platform}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure