Exploring a New Grammatico-functional Type of Measure as Part of a Language Learning Expert System

Cyriel Mallart, Andrew Simpkin, Rmi Venant, Nicolas Ballier, Bernardo Stearns, Jen Yu Li, Thomas GaillatACL 2023

TL;DR

This paper introduces a novel approach to building expert systems for language learning by focusing on "grammatico-functional microsystems" – specific sets of related grammatical forms. By analyzing the probabilistic use of these forms (like IT, THIS, THAT) in learner writing, the authors demonstrate a strong correlation with CEFR proficiency levels, paving the way for more nuanced and interpretable feedback in Intelligent Computer-Assisted Language Learning (ICALL) systems.

What Problem Does This Paper Address?

Current Intelligent Computer-Assisted Language Learning (ICALL) systems often rely on error detection and correction based on native speaker norms. While effective, this approach overlooks the meta-knowledge language learning experts possess about how learners acquire language at different stages. This "expert knowledge" includes understanding the range of linguistic profiles, including hesitations and idiosyncratic uses, that learners exhibit as they progress. The paper aims to bridge this gap by developing a more sophisticated "expert model" that captures these learner-specific linguistic characteristics, moving beyond simple error identification to provide more insightful profiling and feedback.

Methods

The core of the paper's methodology lies in defining and measuring "grammatico-functional microsystems" (MS). These are sets of words that share a similar linguistic function and are in paradigmatic relation (i.e., they are potential substitutes for each other). The authors focus on the IT, THIS, THAT proform microsystem.

They propose three methods to measure the relative use of these forms within their linguistic context:

  1. Proportions: A simple count of the relative frequency of each form within the microsystem in a given text.
  2. Multinomial Logistic Regression: A statistical model that predicts the probability of a specific form being used based on linguistic features extracted from its local context (e.g., part-of-speech, dependency relations).
  3. Neural Network: A more complex model that also predicts the probability of a form's use based on its linguistic environment, leveraging a Leaky ReLU activation function to handle sparse feature representations.

The extracted measures are then used to predict CEFR proficiency levels via ordinal logistic regression.

Datasets

The study utilizes the EFCAMDAT corpus, a large dataset of learner writings collected from EF EnglishTown, an online language school. This corpus contains over 1.18 million texts from students in 191 countries, annotated with 16 proficiency levels that were converted to the six CEFR levels (A1-C2).

Key Insights and Ideas

  • Learner Hesitation as a Feature: The paper frames learner language not just as correct or incorrect, but as a space of choices and hesitations. Microsystems capture these choices, reflecting the learner's internal "competition" between similar forms.
  • Beyond Error Correction: The research advocates for ICALL systems that go beyond simply identifying errors. By modeling the likelihood of using certain forms, systems can gain a deeper understanding of a learner's current proficiency and developmental stage.
  • Context is Crucial: The effectiveness of the probabilistic models (logistic regression and neural network) highlights the importance of the surrounding linguistic context in understanding proform usage.
  • IT, THIS, THAT as a Case Study: The choice of the IT, THIS, THAT proform microsystem is strategic, as these demonstrative pronouns and the pronoun "it" often have overlapping functions but distinct usage patterns across proficiency levels.

Key Results

  • Predictive Accuracy: The multinomial logistic regression achieved an overall accuracy of 0.77 in predicting proform usage based on context, while the neural network achieved 0.74.
  • Proficiency Correlation: All three measure construction methods (proportions, logistic regression, neural network) showed significant associations with CEFR levels (p < 0.001).
  • Differential Performance: The neural network showed better performance for predicting THAT usage, potentially due to its ability to create a high-dimensional feature space that can disentangle the various functions of THAT. The multinomial regression performed better for IT.
  • Proportion Limitations: While proportions showed significant differences across CEFR levels, they were less reliable predictors compared to the probabilistic models, especially for the proform THIS, likely due to its scattered distribution and lack of contextual information.
  • Ordinal Regression Insights: Ordinal logistic regression revealed that higher predicted probabilities of IT usage correlated with lower CEFR levels, while higher probabilities of THAT and THIS usage correlated with higher CEFR levels.

Limitations and Future Work

  • Extraction Challenges: The paper acknowledges that the Grew pattern extraction for the THAT proform needs refinement, as it confused the proform use with the relativizer use.
  • Data Imbalance: The study noted an imbalance in the forms (e.g., THIS proform making up only 2% of samples), which can bias statistical models.
  • Feature Engineering: The authors suggest that incorporating more significant features, such as referential information (degree of givenness), could improve model performance.
  • Advanced NLP: Exploring state-of-the-art NLP approaches like LSTMs and BERT is proposed for better contextualization and potentially more accurate probability predictions.
  • Model Combination: Future work could involve combining the strengths of the different measurement approaches (e.g., through concatenation or weighted sums) to create more robust microsystem measures.
  • Inter-rater Agreement: A future development will include multiple annotators to establish inter-rater agreement for gold standard annotations.

Summary updated

BibTeX
@inproceedings{mallart2023exploring,
  author = {Cyriel Mallart and Andrew Simpkin and Rmi Venant and Nicolas Ballier and Bernardo Stearns and Jen Yu Li and Thomas Gaillat},
  title = {{Exploring a New Grammatico-functional Type of Measure as Part of a Language Learning Expert System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure