The making of the Litkey Corpus, a richly annotated longitudinal corpus of German texts written by primary school children

Ronja Laarmann-Quante, Stefanie Dipper, Eva BelkeACL 2019

TL;DR

This paper introduces the Litkey Corpus, a unique, richly annotated longitudinal dataset of German texts written by primary school children. It addresses the gap in research on first-language literacy acquisition by providing a resource for studying how children learn to spell and write, complete with detailed linguistic annotations and error categorization.

What Problem Does This Paper Address?

Most computational and corpus-based research on language acquisition has focused on second language learners, often adults who have already mastered their first language's orthography. This leaves a significant gap in understanding the complex process of first language literacy acquisition, particularly in children. Specifically, how do young learners, for the first time, grapple with the non-trivial task of mapping spoken sounds to written characters in their native language? Existing corpora of children's texts often correct grammatical errors alongside spelling errors, making it difficult to isolate and study spelling acquisition. This paper aims to fill this void by creating and detailing the Litkey Corpus, a resource specifically designed for this purpose.

Methods

The Litkey Corpus was built upon existing picture story texts written by primary school children. The core methodology involved:

  • Manual Transcription and Annotation: Initial transcription of the children's texts was followed by manual annotation of a "target hypothesis" for each word, correcting spelling errors while preserving grammatical structure. Inter-annotator agreement (IAA) was high for both steps.
  • Semi-Automatic Linguistic Annotation: Leveraging the target forms, the corpus was enriched with various linguistic annotations:
  • Part-of-Speech (POS) Tagging: A specialized POS tagger was trained on the Litkey data due to the known lower performance of standard taggers on non-standard child language.
  • Word-Internal Structure: This involved analyzing:
  • Phonemes and PCUs (Phoneme-Corresponding Units): Using the G2P (Grapheme-to-Phoneme) web service, phonemes and their alignment with characters were extracted. This was followed by manual correction of identified errors.
  • Syllables: Syllable boundaries and stress were assigned, with a "reduced" category introduced for syllables with a specific nucleus.
  • Morphemes: Stems and affixes were identified, with POS tags used to disambiguate certain stem morphemes.
  • Key Orthographic Features (KOFs): A scheme of fine-grained spelling categories was developed to pinpoint locations prone to errors. These were then mapped to broader KOFs that relate to specific spelling principles (e.g., syllabic principles, morpheme constancy).
  • Error Categorization: The corpus includes a categorization of spelling errors, distinguishing between errors that violate KOFs and those that do not.

Datasets

  • Litkey Corpus: This is the primary dataset introduced in the paper. It consists of longitudinal texts from primary school children in Germany (grades 2-4), with a subset of 251 children contributing an average of 7.7 texts each, totaling 1,922 texts and over 212,000 tokens. The corpus is available online.
  • Existing Corpora for Training Data: The Osnabrücker Bildergeschichtenkorpus (Thelen, 2000, 2010) and the H1 Corpus (Berkling, 2016) were used to create initial training data for the POS tagger.
  • BAS G2P Web Service: Used for initial phoneme and syllable analysis.
  • Duden Pronunciation Dictionary: Used as a reference for manual evaluations.
  • Grammar Canoonet: Used as a reference for morpheme analysis.

Key Insights and Ideas

  • The Uniqueness of Child L1 Acquisition: The paper highlights that studying how children learn to spell in their first language is fundamentally different from studying second language acquisition, especially when dealing with the initial mapping of sounds to letters.
  • The Power of Rich Annotation: By annotating at multiple linguistic levels (phonemes, syllables, morphemes, orthographic features, and error types), the Litkey Corpus provides a granular view of spelling development. This allows for more nuanced research questions than previously possible.
  • Statistical Cues in Spelling: The authors suggest that children might implicitly learn spelling patterns from statistical regularities in language (e.g., bigram frequencies, syllable frequencies, orthographic consistency), which can be studied using this corpus.
  • Distinguishing Spelling vs. Grammatical Errors: A key design choice was to correct only spelling errors in the target forms, allowing researchers to isolate and analyze spelling acquisition without the confounding factor of grammatical errors.
  • The Challenge of Non-Standard Data: The paper emphasizes the difficulty of applying standard NLP tools to children's writing, necessitating the development of specialized models and training data.

Key Results

  • High Annotation Accuracy: Comprehensive evaluations demonstrated high accuracy across all annotation levels:
  • POS tagging: 92.81% accuracy.
  • PCUs/Phonemes: 96.19% correct alignment.
  • Syllables: 91.84% correct analysis.
  • Morphemes: 82.88% correct analysis.
  • Key Orthographic Features (KOFs): High accuracy for most features, with specific challenges for "doubleC_other."
  • Error Analysis: The corpus allows for detailed analysis of spelling errors, categorizing them by KOF violations and identifying common error types.
  • Resource Availability: The Litkey Corpus is freely available, making it a valuable resource for the research community.

Limitations and Future Work

  • Limitations:
    • The POS tagger's training data was relatively small (< 110,000 tokens) compared to standard German corpora.
  • Some morpheme analyses were not context-dependent, potentially leading to inaccuracies for ambiguous stems.
  • The evaluation of KOFs was type-based, which sometimes led to incorrect POS tags and subsequent morpheme analyses.
  • The paper notes that some homographs with differing pronunciations required manual correction based on the most common usage, as context was not always available during the correction phase.
  • Future Work:
    • Enriching the corpus with annotations for grammatical errors.
  • Exploring the merging of the Litkey Corpus with larger standard German corpora for improved POS tagger training.
  • Investigating the impact of sentence boundary detection on annotation quality.

Summary updated

BibTeX
@inproceedings{laarmannquante2019making,
  author = {Ronja Laarmann-Quante and Stefanie Dipper and Eva Belke},
  title = {{The making of the Litkey Corpus, a richly annotated longitudinal corpus of German texts written by primary school children}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure