Harnessing GPT to Study Second Language Learner Essays: Can We Use Perplexity to Determine Linguistic Competence?

Ricardo Muñoz Sánchez, Simon Dobnik, Elena VolodinaNAACL 2024

TL;DR

This paper explores whether the perplexity of a large language model (GPT-SW3) on second language (L2) learner essays can serve as a proxy for linguistic competence. The authors found an inverse relationship between CEFR level and perplexity, suggesting that lower perplexity indicates higher proficiency, though it's not a perfect predictor. They also observed that normalizing essays (correcting errors and non-standard language) significantly reduces perplexity, highlighting the impact of linguistic errors on model surprise.

What Problem Does This Paper Address?

Understanding and accurately assessing second language (L2) learner proficiency is crucial for educational and professional opportunities. Traditional methods often rely on human graders, which can be time-consuming and subjective. While large language models (LLMs) like GPT have shown remarkable capabilities in various NLP tasks, their "black box" nature makes it difficult to understand why they make certain predictions or how they process L2 learner language. This paper aims to bridge this gap by investigating whether a quantifiable metric like perplexity, derived from an LLM's prediction of an L2 essay, can reliably reflect the learner's linguistic competence. The authors specifically focus on Swedish L2 learners, addressing a gap in research that has predominantly used English.

Methods

The core methodology involves using a decoder-only LLM, specifically GPT-SW3, to calculate the perplexity of L2 Swedish learner essays. Perplexity, in this context, is interpreted as a measure of how "surprised" the model is by the sequence of words in an essay. A lower perplexity suggests the essay's language is more predictable by the model, and thus, potentially more linguistically competent.

Two main experimental setups were employed:

  1. Perplexity and CEFR Levels: Essays were analyzed at both an essay-wide and token-level to correlate perplexity scores with the Common European Framework of Reference for Languages (CEFR) levels assigned to the essays.
  2. Perplexity and Text Normalization: The study compared the perplexity of original L2 learner essays with their normalized versions (corrected for errors and non-standard language) to understand the impact of linguistic deviations on perplexity.

Datasets

  • Swell Corpus Collection: This is the primary dataset used, comprising L2 learner essays in Swedish. It is divided into two parts:
  • Swell-pilot: Contains 502 essays annotated with CEFR levels, used for the first experiment.
  • Swell-gold: Contains 502 essays with both original and normalized versions, and proficiency levels based on course enrollment (not direct performance assessment), used for the second experiment.
  • GPT-SW3: This is the language model used for perplexity calculations. It's an autoregressive model trained on a large dataset of Nordic languages.

Key Insights and Ideas

  • Perplexity as a Competence Indicator: The study suggests a general inverse relationship: as L2 learner proficiency (CEFR level) increases, the perplexity of their essays tends to decrease. This implies that more proficient learners produce language that is more predictable to the LLM.
  • Normalization Reduces Surprise: Normalizing L2 essays (correcting errors, non-standard grammar, etc.) significantly lowers their perplexity. This strongly indicates that linguistic errors and non-standard language are major contributors to LLM "surprise."
  • Errors are Key Drivers of Perplexity: The qualitative analysis revealed that a substantial portion of high-perplexity tokens in L2 essays are directly linked to errors, including misspellings, morphological issues, incorrect word choice, and non-idiomatic expressions.
  • Placement Matters (Somewhat): While initial hypotheses about sentence-start perplexity weren't strongly supported, the study noted that perplexity generally tends to decrease towards the end of an essay, possibly due to the model's increasing contextual understanding.
  • Content Words are More Perplexing: Content words (nouns, verbs, adjectives, adverbs) are more frequently associated with high perplexity compared to function words, suggesting that semantic and contextual factors play a larger role in model surprise.

Key Results

  • Quantitative:
    • Essays from higher CEFR levels (B2, C1, C2) generally exhibited lower mean and median perplexity scores compared to lower levels (A1, A2). For instance, A1 essays had a mean perplexity of 5.25, while C2 essays had a mean of 3.46 (Table 3).
  • Normalized essays consistently showed lower perplexity than their original versions across all proficiency levels. The difference in perplexity between original and normalized essays was always positive, confirming the hypothesis. For example, the mean difference for Beginner level was 1.07 (Table 6).
  • Qualitative:
    • High perplexity spikes were often correlated with specific errors like misspellings, incorrect word order, and non-idiomatic phrases (e.g., "efter allt" in Swedish).
  • Punctuation, particularly citation marks and apostrophes (which are non-standard in Swedish), also contributed to perplexity spikes.
  • Proper nouns and names showed varying perplexity, with some being significantly more perplexing than others, suggesting potential biases in the model or data.

Limitations and Future Work

  • CEFR Level as a Proxy: The study acknowledges that proficiency levels derived from course enrollment (Swell-gold) might not perfectly reflect actual essay performance, leading to less clear patterns in Experiment 1.
  • Overlap in Perplexity Scores: Despite the general trend, there was significant overlap in perplexity scores between adjacent CEFR levels, meaning perplexity alone is not a definitive indicator for determining a precise CEFR level.
  • "Surprisal" as a Metaphor: The authors emphasize that interpreting perplexity as "surprisal" is a metaphor, not an indication of the model's sentience or human-like understanding.
  • Future Work:
    • Further exploration is needed to disentangle L2-specific perplexity effects from general linguistic complexity.
  • Investigating the impact of demographic biases in data and models on perplexity scores.
  • Exploring how perplexity scores can be used to place essays on a continuous scale within levels, rather than just discrete categories.
  • A more systematic analysis of word frequencies against perplexity is suggested.

Summary updated

BibTeX
@inproceedings{sanchez2024harnessing,
  author = {Ricardo Muñoz Sánchez and Simon Dobnik and Elena Volodina},
  title = {{Harnessing GPT to Study Second Language Learner Essays: Can We Use Perplexity to Determine Linguistic Competence?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure