Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays

Zixuan Ke, Hrishikesh Inamdar, Hui Lin, Vincent NgACL 2019 · DOI

Introduction

Automated Essay Scoring (AES) systems have traditionally focused on providing a single holistic score. While efficient for large-scale assessments, these scores offer little pedagogical value to students seeking to improve their writing. Researchers from the University of Texas at Dallas are addressing this gap by shifting the focus toward dimension-specific scoring, specifically targeting the "strength" of thesis statements.

The Challenge of Automated Feedback

A strong thesis statement serves as the foundation of a persuasive essay. However, existing AES systems lack the granularity to explain why a thesis is effective or how it can be improved. To bridge this gap, a large-scale, annotated dataset is required to train models capable of recognizing the nuanced attributes that contribute to a compelling argument.

A Multi-Dimensional Annotation Framework

The researchers developed a comprehensive framework to evaluate thesis statements across multiple dimensions. This involved a two-tiered scoring system: a 6-point scale for overall thesis strength and a 3-point scale for ten specific supporting attributes, such as Arguability, Specificity, Clarity, and Relevance.

The 6-point rubric used to evaluate thesis strength.

The study utilized 1,021 persuasive essays from the International Corpus of Learner English (ICLE). Three human annotators were tasked with identifying thesis statements and applying the rubric. Interestingly, the data showed that thesis statements are not always single sentences; they can span multiple sentences or be entirely absent, complicating the task of automated identification.

Distribution of the number of sentences identified as part of the thesis statement across the 1,021 essays.

Key Insights: What Makes a Thesis Strong?

The annotation process revealed several critical insights into the nature of student writing:

  • Relevance is Paramount: "Relevance to Prompt" showed the strongest correlation (0.712) with overall thesis strength. A well-written thesis that fails to address the prompt is fundamentally weak.
  • Strength vs. Clarity: The researchers noted that clarity does not equate to strength. A statement can be perfectly clear but fail as a persuasive thesis if it lacks a controversial claim or a "roadmap" for the essay's development.
  • The "Roadmap" Effect: Attributes like "Direction of Development" (indicating how the essay will unfold) showed high annotator agreement, suggesting these are concrete elements that can be explicitly taught.

Experimental Findings and Model Performance

The researchers established baselines using Linear SVMs and Bi-LSTM Neural Networks to test two primary tasks: identification and scoring.

  • Identification Difficulties: Automated systems struggled to locate thesis statements within essays, achieving a peak F1 score of only 24.5%. This underscores the non-standardized nature of student writing, as shown in the sentence distribution data.
  • Scoring Nuance: When provided with human-identified thesis statements, the best model achieved a Pearson correlation of 0.390 with human scores. While promising, it indicates that "strength" remains a complex quality for current NLP models to capture.
  • Predictive Power of Attributes: Interestingly, when human-labeled attributes were used as features, model performance jumped to a correlation of 0.758, confirming that these ten attributes are indeed the building blocks of a strong thesis.

Future Directions in Automated Writing Evaluation

The study highlights the limitations of current "pipeline" approaches, where errors in identification propagate to scoring. Future work will focus on improving discourse mapping for better thesis identification and developing joint models that identify and score thesis statements simultaneously. By decomposing thesis quality into actionable attributes, this research paves the way for AES systems that function more like writing tutors than simple grading machines.

Summary updated

BibTeX
@inproceedings{ke2019give,
  author = {Zixuan Ke and Hrishikesh Inamdar and Hui Lin and Vincent Ng},
  title = {{Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1390},
}
Figure