On the Effectiveness of Curriculum Learning in Educational Text Scoring

Zijie Zeng, Dragan Gasevic, Guanliang ChenAAAI 2023 · OpenReview

Abstract

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.

1. Introduction

Automatic Text Scoring (ATS) is a high-stakes task in education, helping instructors manage massive workloads in MOOCs or providing instant feedback to students. Traditionally, researchers have tried to improve ATS by building more complex neural network architectures, such as specialized BERT variants. However, this paper argues that the training process itself has been neglected. Grading is inherently difficult; even human experts often disagree on a score. If humans learn better by starting with simple concepts before tackling complex ones, it follows that machine learning models might benefit from a similar "curriculum."

2. Methodology

The study investigates whether Curriculum Learning (CL)—sorting training data by difficulty—can help models generalize better in the messy world of student-authored text. The approach centers on two core components:

2.1 Difficulty Measurers

  • Pre-defined: Human-intuitive metrics including text length, "Distinct-1" (ratio of unique words), Readability (Flesch Reading Ease), and the number of grammatical errors.
  • Automatic: Data-driven metrics where "difficulty" is determined by the model's loss. The authors tested a Static version (fixed ratio) and an Adaptive version (adjusting based on previous epoch performance).

2.2 Training Schedulers

The authors utilized a linear continuous scheduler, gradually increasing the proportion of the dataset available to the model over time. Crucially, they tested both Curriculum (easy-to-hard) and Anti-Curriculum (hard-to-easy) paradigms to determine which approach yields superior performance.

3. Experimental Setup

The study utilized two industry-standard benchmarks released by the Hewlett Foundation:

  • ASAP-SAS: Approximately 17,000 short-answer responses across 10 prompts (Science, Biology, English).
  • ASAP-AES: Approximately 13,000 long-form essays across 8 prompts.

These datasets include scores from two independent human graders, highlighting the inherent subjectivity of the task, with human disagreement rates ranging from 12.8% to 37.9%.

4. Key Results and Discussion

The primary metric for evaluation was the Quadratic Weighted Kappa (QWK), which measures the agreement between the AI and human graders.

Table 1: Performance (QWK) on the ASAP-SAS dataset across various curriculum learning configurations.

For short answers (ASAP-SAS), the Static Automatic measurer achieved the highest average improvement, jumping from a baseline of 0.629 to 0.657, representing a 4.5% relative boost.

Table 2: Performance (QWK) on the ASAP-AES dataset across various curriculum learning configurations.

In contrast, improvements for long essays (ASAP-AES) were more modest. The best performer was the Readability measurer using an Anti-Curriculum approach, moving from 0.735 to 0.743. This suggests that for extremely complex tasks, the "difficulty" of a text is multi-dimensional and harder to capture with simple heuristics.

4.1 Insights on Difficulty Metrics

Among the pre-defined measures, the number of grammatical errors (scaled by length) was the most robust indicator of difficulty. Models trained using this metric consistently outperformed those using length or readability. Interestingly, the "Anti-Curriculum" (hard-to-easy) approach often performed as well as or better than the standard curriculum, suggesting that what humans perceive as "easy" (short, grammatically correct text) might not always be the most informative starting point for a transformer model.

5. Conclusion and Future Work

Curriculum Learning provides a model-agnostic benefit to ATS systems, offering "free" improvements by simply changing the order of training data. While effective, this study was limited to linear schedulers. Future work should investigate more complex pacing functions and utilize attention-weight visualization to understand how a model's focus shifts as it progresses through a curriculum. Furthermore, capturing higher-order cognitive features like logic and argumentation remains an open challenge for defining essay difficulty.

Summary updated

Abstract

Automatic Text Scoring (ATS) is a widely-investigated task in education. Existing approaches often stressed the structure design of an ATS model and neglected the training process of the model. Considering the difficult nature of this task, we argued that the performance of an ATS model could be potentially boosted by carefully selecting data of varying complexities in the training process. Therefore, we aimed to investigate the effectiveness of curriculum learning (CL) in scoring educational text. Specifically, we designed two types of difficulty measurers: (i) pre-defined, calculated by measuring a sample's readability, length, the number of grammatical errors or unique words it contains; and (ii) automatic, calculated based on whether a model in a training epoch can accurately score the samples. These measurers were tested in both the easy-to-hard to hard-to-easy training paradigms. Through extensive evaluations on two widely-used datasets (one for short answer scoring and the other for long essay scoring), we demonstrated that (a) CL indeed could boost the performance of state-of-the-art ATS models, and the maximum improvement could be up to 4.5%, but most improvements were achieved when assessing short and easy answers; (b) the pre-defined measurer calculated based on the number of grammatical errors contained in a text sample tended to outperform the other difficulty measurers across different training paradigms.
BibTeX
@inproceedings{zeng2023effectiveness,
  author = {Zijie Zeng and Dragan Gasevic and Guanliang Chen},
  title = {{On the Effectiveness of Curriculum Learning in Educational Text Scoring}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=DpiMdO2Iz1},
}
Figure