ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring

Shengjie Li, Vincent NgNAACL 2024 · DOI

TL;DR

This paper introduces ICLE++, a new corpus of persuasive essays annotated with both holistic quality scores and fine-grained trait-specific scores. This resource aims to address limitations in existing datasets, particularly regarding the generalizability of automated essay scoring (AES) models and the need for more detailed feedback beyond a single overall score.

What Problem Does This Paper Address?

Automated Essay Scoring (AES) has made significant strides, but current research heavily relies on the ASAP corpus. This reliance presents several challenges:

  • Generalizability: Models trained on ASAP, which consists of essays by native English-speaking middle and high school students, may not perform well on essays from different demographics or writing contexts (e.g., English as a Second Language learners, university students).
  • Limited Feedback: Holistic scores, while useful for overall quality assessment, offer little insight into why an essay received a particular score. This lack of fine-grained feedback hinders students' ability to improve specific writing skills.
  • Cross-Prompt Scoring: Evaluating AES models on essays written for prompts not seen during training is a more realistic scenario, but existing datasets and evaluation methodologies for this task have limitations.

The authors propose ICLE++ to tackle these issues by providing a more diverse and richly annotated dataset.

Methods

The core contribution is the ICLE++ corpus. The authors meticulously collected and annotated this corpus:

  • Corpus Collection: Essays were selected from the International Corpus of Learner English (ICLE), specifically focusing on persuasive essays written by university undergraduates from 16 countries. This provides a contrast to ASAP's native English-speaking student population.
  • Annotation Scheme:
    • Overall Quality: Scored on a 1-4 scale with half-point increments, similar to the GRE Argument task.
  • Trait Scoring: Ten fine-grained traits were identified in consultation with instructors, covering aspects like Prompt Adherence, Thesis Clarity, Argument Persuasiveness, Development, Coherence, Cohesion, Sentence Structure, Vocabulary, and Technical Quality. Each trait is scored on a 1-4 scale with half-point increments.
  • Inter-Annotator Agreement: The authors measured agreement using Krippendorff's alpha and Quadratic Weighted Kappa, reporting substantial agreement for both overall quality and individual traits, indicating the reliability of their annotation process.

Datasets

  • ICLE++ Corpus: A newly introduced dataset of 1008 persuasive essays written by university undergraduates from 16 countries. Essays are annotated with holistic scores and 10 fine-grained trait scores.
  • ASAP Corpus: Used for comparison and benchmarking. The paper notes limitations of ASAP, including its focus on native English speakers and potentially coarse-grained trait annotations.

Key Insights and Ideas

  • Trait Granularity Matters: The paper argues that ASAP's "CONTENT" trait is too coarse-grained, lumping together distinct aspects like thesis clarity and argument persuasiveness. ICLE++'s finer-grained traits offer more actionable feedback.
  • Generalizability is a Challenge: Experiments show that AES models trained on ASAP do not always perform as well on ICLE++, highlighting the need for diverse training data and robust models that can generalize across different writing styles and demographics.
  • Trait Importance Varies: Correlation analysis reveals that while some traits (like Argument Persuasiveness and Development) are strongly correlated with overall quality, others (like Thesis Clarity) have a weaker correlation, suggesting different levels of impact on essay quality.
  • Readability Features Differ in Importance: Interestingly, readability features show a stronger correlation with overall quality in ICLE++ than in ASAP, suggesting that readability might be a more critical factor for non-native English learners.

Key Results

  • Holistic Scoring: Models trained on ICLE++ generally achieve lower QWK scores compared to ASAP, indicating ICLE++ is a more challenging corpus. The inclusion of traits shows mixed results for within-prompt scoring but a generally positive impact on cross-prompt scoring.
  • Trait Scoring: Results for trait-specific scoring on ICLE++ are poorer than on ASAP, which the authors suggest might explain why incorporating traits doesn't always improve holistic scoring on ICLE++.
  • Feature Analysis: The study found that prompt-independent features are more useful for ASAP than for ICLE++, and that features with the strongest correlation to overall quality differ between the two corpora.

Limitations and Future Work

  • Persuasive Essays Only: The current ICLE++ corpus is limited to persuasive essays. The authors believe their framework can be extended to other essay types.
  • Non-Native English Speakers: While this is a strength for generalizability, it means the findings might not directly translate to native English speakers.
  • Model Performance on ICLE++: The poorer performance of existing AES models on ICLE++ suggests that new approaches are needed to effectively model the nuances of essays written by diverse learners. Future work could focus on developing models that better capture the fine-grained traits and generalize across different writing styles and prompts.

Summary updated

BibTeX
@inproceedings{li2024icle,
  author = {Shengjie Li and Vincent Ng},
  title = {{ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.468},
}
Figure