Can GPT-4 do L2 analytic assessment?

Stefano Bannò, Hari K. Vydana, Kate M. Knill, Mark J. F. GalesNAACL 2024

TL;DR

This paper explores whether GPT-4 can perform "analytic scoring" for L2 English writing, breaking down essays into specific proficiency components rather than just giving an overall score. The study finds that GPT-4, even in a zero-shot setting, can extract meaningful analytic information, showing promising correlations with linguistic features, suggesting a potential for more nuanced automated feedback.

What Problem Does This Paper Address?

Automated Essay Scoring (AES) has been around for decades, with significant advancements in "holistic scoring" (an overall score) that can match or even exceed human performance. However, "analytic scoring" – evaluating specific aspects like grammar, vocabulary, and coherence separately – remains a challenge. Human analytic scoring is time-consuming and prone to inconsistencies, making it difficult to obtain reliable ground truth data. Existing AES systems have also struggled with this granular evaluation. The advent of Large Language Models (LLMs) like GPT-4 presents a new opportunity to tackle this problem, potentially automating the detailed feedback that L2 learners need for improvement.

Methods

The researchers employed a two-stage approach using GPT-4. First, they trained a Longformer-based model to predict holistic CEFR scores for L2 English essays. This model was trained on the EFCAMDAT dataset and fine-tuned on the W&I dataset. Second, they used GPT-4 in a zero-shot setting to perform analytic scoring. This involved prompting GPT-4 with an essay, its holistic score (either ground truth or predicted), and specific CEFR descriptors for a particular analytic aspect. GPT-4 was asked to select the descriptor that best matched the essay. They also conducted experiments without providing the holistic score to GPT-4 to assess its impact.

Datasets

  • Write & Improve (W&I): An online platform for L2 English learners. This dataset contains approximately 3,300 compositions manually annotated with CEFR levels. For this study, the CEFR levels were mapped to a 1-6.5 scale.
  • EFCAMDAT: A large publicly available L2 learner corpus containing over 1.18 million scripts from over 174,000 learners, annotated with scores from 0-100 and proficiency levels (mapped to CEFR). A subset of 753,508 essays was used for training and 7,612 for validation.

Key Insights and Ideas

  • GPT-4's Zero-Shot Capability: Even without specific training on analytic scoring tasks, GPT-4 demonstrated an ability to extract meaningful analytic information from essays. This suggests LLMs can be powerful tools for fine-grained assessment.
  • Holistic Score as a Guide: Providing GPT-4 with the holistic score significantly improved its ability to perform analytic scoring, acting as a crucial guide for the model. This highlights the interconnectedness of different proficiency aspects.
  • Feature Correlation: The study found significant correlations between GPT-4's predicted analytic scores and various linguistic features extracted from the essays (e.g., grammatical error rate, unique word count). This provides evidence for the validity of GPT-4's assessments and offers insights into what features contribute to different proficiency aspects.
  • Grammar and Vocabulary Strengths: GPT-4's performance was particularly strong for grammar and vocabulary-related analytic scores, aligning well with traditional feature-based approaches.
  • Pragmatic Competence Nuances: While generally successful, some pragmatic competence aspects (like flexibility) showed less clear correlations, potentially due to their "longitudinality" or the complexity of their descriptors.

Key Results

  • GPT-4 achieved a significant correlation of 0.797 with holistic scores even when not provided with any explicit holistic score information.
  • When provided with holistic scores (either ground truth or predicted), GPT-4's correlation with the average of predicted analytic scores reached 0.904 (with ground truth holistic scores) and 0.898 (with predicted holistic scores).
  • Grammatical error rate (%gram.) showed the highest correlations with grammatical accuracy and orthographic control aspects.
  • The number of unique difficult words (#dif.wds.) was a strong indicator for vocabulary control.
  • Statistical tests (Friedman test and post-hoc Nemenyi test) indicated significant differences among the predicted analytic scores, suggesting GPT-4 can differentiate between these aspects.

Limitations and Future Work

  • Lack of Ground Truth Analytic Scores: The primary limitation is the absence of human-annotated analytic scores for the W&I dataset, making direct validation of GPT-4's analytic scores challenging. The authors acknowledge that human analytic scoring itself can be unreliable.
  • Dataset Specificity: The study used specific datasets (W&I and EFCAMDAT) and a particular LLM (GPT-4). Generalizability to other datasets or LLMs requires further investigation.
  • Future Work: The authors plan to deploy this system in educational settings to evaluate its effectiveness by monitoring learner progress. They also intend to explore multi-modal systems for assessing L2 speech. Further research will focus on understanding the nuances of pragmatic competence assessment and exploring the impact of different LLMs and prompting strategies.

Summary updated

BibTeX
@inproceedings{banno2024can,
  author = {Stefano Bannò and Hari K. Vydana and Kate M. Knill and Mark J. F. Gales},
  title = {{Can GPT-4 do L2 analytic assessment?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure