Automatic Task Requirements Writing Evaluation via Machine Reading Comprehension

Shiting Xu, Guowei Xu, Peilei Jia, Wenbiao Ding, Zhongqin Wu, Zitao Liu

Abstract

Traditional Automated Essay Scoring (AES) systems often provide holistic scores that lack interpretability, failing to pinpoint whether specific task requirements have been met. This paper presents an end-to-end framework leveraging Machine Reading Comprehension (MRC) to evaluate task requirement fulfillment in student writing. By treating requirements as questions and essays as context, the system achieves an accuracy of 0.93 and an F1 score of 0.85. The approach incorporates Question Normalization and a dual-verification strategy to bridge the gap between prompt instructions and student responses, providing actionable feedback through response locating.

Introduction

In English proficiency examinations such as the Key English Test (KET) and Preliminary English Test (PET), writing tasks are evaluated based on how well students address specific "Task Requirements" (TRs). For instance, a prompt might ask a student to explain a choice, identify participants, and state a time. While current AES systems can provide a numerical grade, they often function as "black boxes," offering little insight into which specific requirements were missed. This lack of granularity hinders effective pedagogical feedback. To address this, researchers have developed a framework that utilizes MRC to verify and locate responses within student essays.

Methodology

The proposed framework consists of three primary modules designed to handle the linguistic nuances of student writing and the structural requirements of MRC.

Overview of the task requirement evaluation framework, showing the flow from requirement normalization to response locating.
  1. Question Normalization (QN): There is often a perspective mismatch between prompts (written by examiners using "you") and essays (written by students using "I"). The QN module uses rule-based methods to align these perspectives and remove instructional filler (e.g., "tell your friend"), ensuring the "question" more closely matches the "context."
  2. MRC Module: Built upon the ELECTRA language model, this module processes the normalized requirement and essay. It employs a dual-verification strategy to determine if a requirement is answered.
  3. Response Locating (RL): This module identifies the specific token sequence in the essay that satisfies the requirement. If the start/end positions are contradictory, the requirement is marked as unmet.
Detailed architecture of the ELECTRA-based MRC module, featuring span prediction and dual-verification components (E-FV and TAV).

As shown in Figure 2, the MRC architecture utilizes an ELECTRA discriminator as its backbone. It includes a Span Prediction layer for identifying start and end positions and an Answerable Verification layer. The latter combines External Front Verification (E-FV), which analyzes the [CLS] token, and Threshold-based Answerable Verification (TAV), which evaluates the probability of "no-answer" based on span scores.

Datasets and Training

The model was evaluated using two datasets:

  • SQuAD 2.0: A general-purpose benchmark used for initial "warm-up" training to teach the model basic reading comprehension mechanics.
  • Student Essay Dataset (SED): A specialized dataset of 12,807 real-world student examples. SED is characterized by longer answer spans (averaging 103 characters) and frequent grammatical errors, presenting a significant domain gap from standard benchmarks.

Key Results and Insights

  • Top Performance: The ELECTRA-based framework achieved a 0.93 accuracy and 0.85 F1 score on the SED test set, significantly outperforming BERT, RoBERTa, and ALBERT baselines.
  • The Power of Fine-tuning: A key finding was the necessity of domain-specific fine-tuning. Models trained only on SQuAD performed poorly on student essays due to the "perspective gap" and grammatical errors. Fine-tuning on SED resulted in a 23% improvement in accuracy.
  • Interpretability: Unlike "black box" scorers, this system can highlight the exact sentence that satisfied a requirement, making it a viable tool for providing actionable classroom feedback.

Conclusion

By reframing task requirement evaluation as an MRC problem, this research provides a path toward more interpretable automated grading. The ability to highlight exact sentences that satisfy specific prompts allows for more meaningful feedback, helping students understand precisely where their writing succeeds or fails. Future work will focus on refining the model's sensitivity to the quality of responses beyond simple binary verification.

Summary updated

BibTeX
@inproceedings{xu2021automatic,
  author = {Shiting Xu and Guowei Xu and Peilei Jia and Wenbiao Ding and Zhongqin Wu and Zitao Liu},
  title = {{Automatic Task Requirements Writing Evaluation via Machine Reading Comprehension}},
  booktitle = {AIED},
  year = {2021},
  url = {https://openreview.net/forum?id=13UPcbp1If},
}
Figure