Recognizing Learner Handwriting Retaining Orthographic Errors for Enabling Fine-Grained Error Feedback

Christian Gold, Ronja Laarmann-Quante, Torsten ZeschACL 2023

TL;DR

This paper tackles the challenge of providing automated feedback on spelling errors in handwritten learner texts by developing a handwriting recognition system that preserves these errors. They introduce a new dataset of transcribed learner handwriting and demonstrate that their specialized HWR system significantly outperforms standard approaches, paving the way for more effective educational tools.

What Problem Does This Paper Address?

Providing automated feedback on student writing is a valuable educational tool, but it's significantly hampered when dealing with handwritten work. While current handwriting recognition (HWR) systems are highly accurate, they typically "normalize" away orthographic errors (like misspellings) to produce clean text. This normalization, while useful for general transcription, removes the very information needed to identify and correct a learner's specific spelling mistakes. The core problem is bridging the gap between an image of handwritten text containing errors and a digital transcript that retains those errors for analysis.

Methods

The authors propose a two-pronged approach:

  1. Dataset Creation: They created a novel dataset by transcribing 1,350 pages of German learner handwriting from the FD-LEX corpus. Crucially, their transcription guidelines were designed to meticulously preserve orthographic errors, strikethroughs, and other linguistic nuances, unlike standard HWR datasets.
  2. Specialized HWR System: They trained a simple HWR system using a Convolutional Neural Network (CNN) with Connectionist Temporal Classification (CTC) decoding. The key innovation here is not just the architecture but how it's trained and evaluated:
  3. Error-Retaining Transcripts: The system is trained on their newly created dataset, which includes orthographic errors.
  4. Dictionary Constraint (WBS): They employ Word Beam Search (WBS) to constrain the output. Instead of using standard dictionaries of correctly spelled words, they experiment with custom dictionaries:
  5. childLex: A German dictionary derived from children's books.
  6. FD-LEX Dictionary: A dictionary built from the normalized transcripts of their own dataset.
  7. Spelling Error Generator: They developed a system to generate systematic spelling variants of words from these dictionaries, mimicking common learner errors. This allows them to create "expanded" dictionaries that include plausible misspellings.

Datasets

  • FD-LEX: The source corpus for learner handwriting. The authors created new, error-retaining transcripts for 1,350 pages from this dataset.
  • IAM & CVL: Standard English handwriting datasets used for initial baseline training and comparison.
  • childLex: A German dictionary derived from children's books, used for WBS.
  • FD-LEX Dictionary: A dictionary compiled from the normalized transcripts of the FD-LEX corpus.

The primary contribution is the newly transcribed FD-LEX dataset that preserves orthographic errors.

Key Insights and Ideas

  • The Value of "Bad" Data: Standard HWR systems are optimized for accuracy by removing errors. However, for educational feedback, these "errors" are precisely the data of interest. This paper highlights the need for specialized HWR systems tailored to specific downstream tasks.
  • Dictionary Design is Crucial: Simply using a standard dictionary with WBS can eliminate spelling errors. The real gains come from using dictionaries that include plausible learner errors, either from existing learner corpora or generated systematically.
  • Systematic Error Generation is Effective: Learners often make predictable types of spelling mistakes. Generating these systematic errors and incorporating them into dictionaries significantly improves HWR performance on learner texts.
  • Learner Texts are Different: The performance differences observed between grades (9th vs. 5th) and school types (GYM vs. IGS) suggest that learner language has distinct characteristics that impact HWR.

Key Results

  • Baseline Performance: A standard HWR setup trained on combined IAM/CVL and their dataset achieved a Word Error Rate (WER) of 37.6% on their test set.
  • Improved Baseline: Training only on their dataset improved the baseline WER to 34.7%.
  • WBS with Custom Dictionaries:
    • Using the childLex dictionary reduced WER to 29.6%.
  • Using the FD-LEX dictionary reduced WER to 31.3%.
  • WBS with Error-Expanded Dictionaries:
    • Combining childLex with generated spelling errors (WBS childLex + SP) achieved a WER of 29.0%.
  • Combining FD-LEX with generated spelling errors (WBS FD-LEX + SP) achieved a WER of 25.9%.
  • The best result was achieved by combining both childLex and FD-LEX dictionaries with spelling errors (WBS childLex + FD-LEX + SP), yielding a WER of 25.0%. This is a significant improvement, reducing the error rate by almost one-third compared to the baseline.
  • Coverage: Even with generated errors, their combined dictionaries covered 90-94% of the test set vocabulary.

Limitations and Future Work

  • Dictionary Size: The generated spelling error dictionaries become extremely large (millions of words), raising questions about their practicality for real-time applications.
  • Model Complexity: The paper focuses on dictionary and data aspects, not on improving the core HWR model itself. Future work could explore more sophisticated models, like full-page recognition.
  • Error Prioritization: The authors suggest that future work should focus on allowing more probable errors while avoiding overwhelming the dictionary. This involves analyzing which errors are most common among learners and which generated errors are truly useful.
  • Targeted Error Generation: Developing more targeted error generation strategies based on common learner error patterns and combinations is a promising direction.

Summary updated

BibTeX
@inproceedings{gold2023recognizing,
  author = {Christian Gold and Ronja Laarmann-Quante and Torsten Zesch},
  title = {{Recognizing Learner Handwriting Retaining Orthographic Errors for Enabling Fine-Grained Error Feedback}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure