Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model

Jakob Prange, Man Ho Ivy WongACL 2023 · DOI

Abstract

The acquisition of English prepositions presents a significant challenge for second-language (L2) learners due to their idiomatic and formulaic nature. Traditional research often relies on aggregate frequentist statistics, which can obscure individual and item-level variations. This study re-analyzes behavioral data from 71 Chinese learners of English using Bayesian Mixed Effects models and the RoBERTa language model. Our results demonstrate that Bayesian models not only replicate previous findings with greater sensitivity but also reveal that the ability to reject ungrammatical usage (specificity) is a distinct and critical component of linguistic competence, more closely linked to production than the ability to accept grammatical usage (sensitivity).

Introduction

Learning prepositions such as in, at, and over is notoriously difficult for L2 learners. Their usage often defies simple rules, leaning instead on formulaic patterns. Conventional Second Language Acquisition (SLA) research typically employs frequentist methods like ANOVA to analyze group-level performance. However, these "macro" analyses fail to explain why specific items or categories (e.g., spatial vs. abstract) pose varying difficulties for different students. This research addresses this gap by applying computational tools to perform a "micro" item-level analysis, modeling the complex interactions between individual student ability, task type, and linguistic properties.

Methodology

We utilized a dataset from Wong (2022) comprising 17,644 responses from 71 native Chinese speakers. The study focused on 12 senses of in, at, and over across Grammaticality Judgment Tests (GJT) and Picture Elicitation Tests (PET). Three computational strategies were employed:

  1. Bayesian Logistic Model (BLM): An Item Response Theory (IRT) inspired approach that treats student ability and item difficulty as latent variables, accounting for random effects of individuals and stimuli.
  2. Multilayer Perceptron (MLP): A 3-layer neural network used as a non-linear predictive baseline.
  3. RoBERTa: A pretrained language model used to calculate contextual probabilities, serving as a proxy for the "formulaicity" of prepositional usage.

Results and Discussion

The Bayesian approach successfully replicated the finding that concept-based instruction is superior to traditional methods while revealing interactions missed by frequentist models. As shown in Figure 2, the Bayesian model ("Ours") consistently identified larger effect sizes ($d$) and higher statistical significance compared to the original frequentist analysis (W22).

Figure 2: Comparison of effect sizes and significance between the original frequentist analysis (W22) and the current Bayesian approach (Ours) across GJT and PET tasks.

A key insight emerged regarding the nature of receptive knowledge. We distinguished between "sensitivity" (GJT-Y: accepting correct items) and "specificity" (GJT-N: rejecting incorrect items). Figure 1 illustrates the relationship between these two skills and production ability (PET). We found that specificity (GJT-N) is more strongly correlated with the ability to produce correct language in the PET than sensitivity is.

Figure 1: Scatter plot showing the relationship between sensitivity (GJT-Y) and specificity (GJT-N), with colors indicating performance on the Picture Elicitation Test (PET).

Furthermore, while RoBERTa's predictions correlated with human judgments ($R^2 = 0.48$), the model acted as a "conservative" judge, excelling at flagging errors but occasionally failing to recognize human-acceptable nuances. Both the Bayesian model and the MLP achieved approximately 72.5% accuracy in predicting individual student responses, but the BLM offered superior interpretability for scientific analysis.

Conclusion

This study highlights the advantages of interpretable Bayesian models over traditional frequentist methods and "black box" neural networks in SLA research. By providing detailed student profiles, these models can inform the development of personalized educational tools that target specific weaknesses, such as a learner's inability to reject ungrammatical structures. Future work should explore "LM probing" to refine test stimuli and further investigate the divergence between human and machine linguistic judgments.

Summary updated

BibTeX
@inproceedings{prange2023reanalyzing,
  author = {Jakob Prange and Man Ho Ivy Wong},
  title = {{Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.712},
}
Figure