QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays

May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud, Houda BouamorACL 2024

TL;DR

This paper introduces QAES, the first publicly available dataset of trait-specific annotations for automated essay scoring (AES) in Arabic. By providing detailed scores for aspects like relevance, organization, and grammar, QAES aims to bridge a significant resource gap and enable the development of more nuanced AES systems for Arabic, moving beyond simple holistic scores.

What Problem Does This Paper Address?

Automated Essay Scoring (AES) is a valuable tool for educators to assess student writing, but its application to Arabic has been hampered by a lack of suitable datasets. While English AES benefits from large, annotated benchmarks, Arabic AES research is significantly behind. Existing Arabic datasets are often small, proprietary, or lack the detailed, trait-specific annotations needed to understand why an essay is good or bad, not just how good it is overall. This limits the ability to provide targeted feedback for improvement. The authors aim to address this by creating QAES, a publicly available, trait-annotated corpus for Arabic essays.

Methods

The core of this work is the creation of the QAES dataset. This involved annotating existing Arabic essays from the Qatari Corpus of Argumentative Writing (QCAW). The annotation process used a rubric adapted from the Core Academic Skills Test (CAST), which assesses seven specific traits: Relevance (REL), Organization (ORG), Vocabulary (VOC), Style (STY), Development (DEV), Mechanics (MEC), and Grammar (GRA), in addition to an overall Holistic (HOL) score. To ensure annotation quality, three annotators were involved, with a third acting as a tie-breaker for disagreements. The paper also details experiments using baseline AES models (feature-based and English SOTA adapted for Arabic) to evaluate the QAES dataset and establish initial performance benchmarks.

Datasets

  • Qatari Corpus of Argumentative Writing (QCAW): This is the source corpus for the QAES dataset. It contains 195 Arabic essays written by first-year university students in response to two distinct argumentative writing tasks.
  • QAES (Qatari Corpus of Argumentative Writing Annotations for Automated Essay Scoring): This is the newly introduced dataset, comprising the QCAW essays with detailed trait-specific and holistic annotations. It is made publicly available to the research community.

Key Insights and Ideas

  • Trait-specific scoring is crucial for actionable feedback: The paper emphasizes that holistic scores alone are insufficient for guiding students. Trait-specific scores offer a much deeper understanding of strengths and weaknesses.
  • Arabic AES is significantly underdeveloped: The lack of publicly available, richly annotated datasets is a major bottleneck for advancing Arabic AES research.
  • Task difficulty impacts annotation and model performance: The study found that Task 1 essays were generally harder to score and led to lower inter-annotator agreement compared to Task 2, suggesting differences in student comprehension or essay quality between the tasks.
  • Feature importance varies by trait and task: An ablation study revealed that different feature categories (e.g., N-grams, semantics) have varying impacts on model performance for different writing traits and tasks, highlighting the need for tailored feature selection.
  • RMSE might be a more reliable metric than QWK for small datasets: For the small QAES dataset, RMSE showed more consistent performance, suggesting it might be a better indicator of model accuracy when sample sizes are limited.

Key Results

  • Inter-Annotator Agreement (IAA): The average Quadratic Weighted Kappa (QWK) scores for annotator agreement were substantial, ranging from 0.677 for Task 1 to 0.778 for Task 2, indicating good consistency.
  • Baseline Model Performance:
    • Holistic Scoring: The Linear Regression (LR) model achieved the highest QWK scores, but also the highest RMSE, suggesting it captured overall patterns but lacked precision. Neural models (STL, MTL) showed lower RMSE, indicating more precise predictions, but generally lower QWK. Overall holistic scoring performance was weak.
  • Trait Scoring: The LR model generally achieved higher QWK scores across traits, indicating better agreement with human annotators. However, STL models often achieved lower RMSE, suggesting higher accuracy in predicting trait scores. MTL models struggled in some traits.
  • Trait Distributions: Task 2 essays generally received higher scores across most traits, indicating a distinction in writing quality or task comprehension between the two tasks. Mechanics (MEC) and Grammar (GRA) scores showed consistency across tasks, suggesting they are less task-dependent.

Limitations and Future Work

  • Dataset Size: The QAES dataset, while valuable, is still relatively small compared to established AES datasets in other languages. This limits the application of complex machine learning models, especially deep learning.
  • Model Specificity: The study focused on adapting existing AES methodologies to Arabic rather than developing models specifically tailored to the unique linguistic characteristics of the language.
  • Annotation Refinement: While substantial, the annotation guidelines could potentially be further refined, and additional moderation sessions might have clarified more ambiguous points.

Future work suggested includes refining annotation guidelines, expanding the dataset, and exploring innovative methodologies for Arabic AES. The authors also encourage research into developing specialized models for Arabic AES that account for the language's unique features.

Summary updated

BibTeX
@inproceedings{bashendy2024qaes,
  author = {May Bashendy and Salam Albatarni and Sohaila Eltanbouly and Eman Zahran and Hamdo Elhuseyin and Tamer Elsayed and Walid Massoud and Houda Bouamor},
  title = {{QAES: First Publicly-Available Trait-Specific Annotations for Automated Scoring of Arabic Essays}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure