Tracking the Evolution of Written Language Competence in L2 Spanish Learners

Alessio Miaschi, Sam Davidson, Dominique Brunato, Felice Dell’Orletta, Kenji Sagae, Claudia Helena Sanchez-Gutierrez, Giulia VenturiACL 2020

TL;DR

This paper introduces an NLP-driven approach to automatically track the development of written Spanish proficiency in second language (L2) learners. By analyzing a rich set of linguistic features from student essays, they demonstrate the ability to predict the chronological order of essays written by the same student at different stages of their learning, offering insights into how explicit instruction aligns with language acquisition.

What Problem Does This Paper Address?

Understanding how individuals learn a second language (L2) and how their proficiency evolves over time is a complex challenge. While traditional methods rely on expert human evaluation, this is time-consuming and can be subjective. Researchers have turned to Natural Language Processing (NLP) to automate the analysis of learner writing, aiming to identify linguistic features that signal progress. However, a significant gap exists in understanding how specific linguistic features change with increasing proficiency and whether these changes directly reflect the curriculum students are exposed to. Existing work often focuses on cross-sectional data or a limited set of features, making it difficult to map the learning journey and connect it to pedagogical interventions. This paper aims to bridge this gap by using a longitudinal corpus of L2 Spanish writing to model development and investigate the relationship between linguistic features and instructional content.

Methods

The core of this research lies in an NLP-based stylometric approach. The authors extract a comprehensive set of linguistic features from student essays, covering various levels of linguistic annotation:

  • Raw Text Properties: Basic text statistics like sentence and word length.
  • POS Tagging: Distribution of parts of speech (nouns, verbs, adjectives, etc.).
  • Dependency Parsing: Analyzing syntactic relationships between words, including features like verb argument structure, subject-object order, and the depth of syntactic trees.
  • Lexical Features: Incorporating word frequency measures to capture lexical richness.

These features are then used to train a Linear Support Vector Machine (LinearSVM). The model is trained on pairs of essays written by the same student at different course levels. The task is framed as a binary classification problem: predicting whether the second essay in a pair represents a higher proficiency level than the first. To gain insights into which features are most predictive, the authors analyze the weights assigned by the LinearSVM, assuming higher weights indicate greater importance in modeling language development.

Datasets

The study utilizes the Corpus of Written Spanish of L2 and Heritage Speakers (COWS-L2H). This corpus comprises 3,498 short essays written by students enrolled in ten lower-division Spanish courses at a single American university. The courses are categorized into beginner, intermediate, composition, and heritage speaker levels. The corpus offers both cross-sectional and longitudinal data, with students submitting multiple compositions over time. The authors also mention using a Spanish Wikipedia dump for calculating word frequencies.

Key Insights and Ideas

  • Predicting Learning Trajectories: The study successfully demonstrates that a wide range of linguistic features can predict the relative order of essays written by the same student at different proficiency levels. This suggests that L2 writing development follows discernible patterns that can be captured computationally.
  • Curriculum Alignment: A key finding is that the most predictive linguistic features often align with the explicit instruction students receive in their courses. This provides empirical evidence for the impact of pedagogical interventions on language acquisition.
  • Beginner Stages Show Strongest Evolution: The most significant improvements in predictive accuracy were observed when comparing essays from the beginner levels (SPA 1, 2, 3). This highlights that the transition from novice to early intermediate writing involves the most dramatic linguistic shifts.
  • Verbal Morphology is Crucial Early On: For beginner learners, features related to verbal morphology, particularly auxiliary verbs, are highly predictive. This makes intuitive sense, as introductory Spanish courses often focus on verb tenses and conjugations.
  • Lexical Density as a Proxy for Complexity: Lexical density, while not always explicitly taught, emerges as an important feature, particularly in the transition from beginner to intermediate levels. This suggests it's a natural consequence of students learning to construct more complex sentences.
  • Advanced Levels Present a Challenge: At more advanced levels (e.g., Composition courses), the predictive power of the current feature set decreases. This might be because these courses focus on discourse-level characteristics and academic writing styles that are not fully captured by the current feature extraction.

Key Results

  • The study achieved document-level classification accuracy of 0.68 (Same prompt) and 0.70 (All prompts) when considering all course level pairs.
  • Sentence-level classification also yielded promising results, with accuracy around 0.59-0.61.
  • The highest accuracy was observed when comparing essays from the beginner levels (e.g., SPA 1-SPA 2 achieved 0.88 accuracy at the document level).
  • Feature ranking analysis revealed that features related to verbal morphology (auxiliary verbs, past tense) were most important for distinguishing between beginner levels, while lexical density and preposition usage became more prominent in later beginner stages.

Limitations and Future Work

  • Advanced Level Limitations: The authors acknowledge that their current feature set may not fully capture the nuances of academic writing and discourse-level characteristics taught in advanced composition courses, leading to lower classification performance at these levels.
  • L1 Influence: The study did not deeply explore the influence of students' first language (L1) on feature rankings. They suggest that L1 transfer and interference effects could significantly impact the rate at which specific linguistic features are acquired.
  • Cross-Lingual Analysis: The authors propose a future direction of conducting a cross-lingual analysis to see if the observed feature rankings in Spanish writing development are similar to those found in the acquisition of other languages.
  • Broader Feature Set: Future work could involve exploring a wider range of features, particularly those related to discourse coherence and pragmatic aspects of writing.

Summary updated

BibTeX
@inproceedings{miaschi2020tracking,
  author = {Alessio Miaschi and Sam Davidson and Dominique Brunato and Felice Dell’Orletta and Kenji Sagae and Claudia Helena Sanchez-Gutierrez and Giulia Venturi},
  title = {{Tracking the Evolution of Written Language Competence in L2 Spanish Learners}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure