A Dependency Treebank of Spoken Second Language English

Kristopher Kyle, Masaki Eguchi, Aaron Miller, Theodore SitherNAACL 2022

TL;DR

This paper introduces the first publicly available dependency treebank of spoken second language (L2) English, enabling better NLP tools for analyzing L2 speech. They demonstrate that including this L2 data in training significantly improves POS tagging and dependency parsing accuracy on L2 texts, especially spoken L2.

What Problem Does This Paper Address?

Natural Language Processing (NLP) tools like part-of-speech (POS) taggers and dependency parsers are invaluable for linguistic research, allowing analysis of large text corpora. However, these tools often perform poorly on "out-of-domain" data, meaning texts that differ significantly from the data they were trained on. This is a major challenge for applied linguists studying second language (L2) English, as existing NLP tools are typically trained on native (L1) English data. While a treebank for written L2 English exists, there's a critical lack of publicly available resources for spoken L2 English. This gap hinders the development and evaluation of NLP models specifically tailored for L2 speech analysis.

Methods

The researchers developed a new dependency treebank of spoken L2 English by annotating a subset of the National Institute of Information and Communications Technology Japanese Learner English (NICT JLE) corpus. This annotation involved both Penn POS tags and Universal Dependencies. To evaluate the impact of this new resource, they trained POS tagger and dependency parser models using the NLP library spaCy (version 3.2). They experimented with three training data configurations:

  1. L1 only: Using only existing L1 English treebanks (EWT, GUM, PUD, UDEP).
  2. L1 + L2: Combining L1 data with their new spoken L2 treebank (SL2E) and the existing written L2 treebank (TLE).
  3. L1 + L2e: Similar to L1+L2, but the total number of tokens was matched to the L1-only training set by subsampling L1 data. This controls for the effect of simply having more data versus having relevant L2 data.

They used pre-trained RoBERTa-base embeddings and optimized models with the Adam optimizer. Accuracy was measured using F1 scores for POS tagging and Labeled Attachment Scores (LAS) for dependency parsing.

Datasets

  • Newly Introduced: Dependency Treebank of Spoken L2 English (SL2E) - annotated with Penn POS tags and Universal Dependencies. This is the core contribution.
  • Existing L1 Datasets (used for training/evaluation):
    • English Web Treebank (EWT)
    • Georgetown University Multilayer Treebank (GUM)
    • Universal Pronouns (UDEP)
    • Parallel Universal Dependencies Treebank (PUD) - used for training only.
  • Existing L2 Dataset (used for training/evaluation):
    • Treebank of Learner English (TLE) - written L2 English.

Key Insights and Ideas

  • The "Out-of-Domain" Problem is Real: NLP models trained solely on L1 data struggle with L2 data, highlighting the need for domain-specific training resources.
  • Spoken L2 English is Underserved: The lack of annotated spoken L2 data is a significant bottleneck for linguistic research.
  • L2 Data Boosts L2 Performance: Including even a relatively small amount of L2 training data (especially spoken L2) significantly improves NLP model performance on L2 texts.
  • Spoken L2 is Particularly Challenging (and Rewarding): The improvements were most pronounced for spoken L2 data, suggesting it's a more difficult domain for standard NLP tools but also where new resources yield the greatest gains.
  • Data Size vs. Data Relevance: The L1+L2e experiment shows that adding relevant L2 data can be more beneficial than simply increasing the total amount of L1 data, especially when token counts are balanced.

Key Results

  • POS Tagging:
    • Models trained on L1 data alone achieved high F1 scores (0.958-0.977) on L1 corpora (EWT, GUM).
  • The lowest accuracy for L1-trained models was on the spoken L2 test set (F1 = 0.936).
  • Including L2 data (L1+L2 and L1+L2e) improved L2 spoken test set accuracy to 0.970, comparable to L1 corpora.
  • Dependency Parsing:
    • Labeled Attachment Scores (LAS) ranged from 0.876 (L1 model on SL2E) to 0.938 (L1+L2e model on SL2E).
  • Accuracy consistently increased with the inclusion of L2 data in the training set for both written and spoken L2 data.
  • The highest dependency annotation accuracy was observed with L2 spoken data.

Limitations and Future Work

  • Limited Gold Standard Data: The total amount of publicly available gold standard data for Universal Dependencies remains relatively small. Future work should focus on creating more such data across diverse English domains, including L2.
  • Granularity of Analysis: While overall F1 scores are useful, future research should delve into the strength and weaknesses of models for specific grammatical features (e.g., verb-object pairs) rather than just aggregate scores.
  • Future Directions: The authors suggest expanding the creation of gold standard data for L2 domains and conducting more fine-grained analyses of model performance on specific linguistic phenomena.

Summary updated

BibTeX
@inproceedings{kyle2022dependency,
  author = {Kristopher Kyle and Masaki Eguchi and Aaron Miller and Theodore Sither},
  title = {{A Dependency Treebank of Spoken Second Language English}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure