Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays

Zarah Weiss, Anja Riemenschneider, Pauline Schröter, Detmar MeurersACL 2019

TL;DR

This paper investigates how language complexity and accuracy influence human grading of German essays, particularly in the context of the high-stakes Abitur exam. Using computational linguistics, they developed a method to identify "task-appropriate" language complexity and found that while teachers can recognize it for language performance, they are heavily biased by accuracy, even when it's irrelevant to content grading.

What Problem Does This Paper Address?

Automated essay scoring (AES) is a growing field, but its application in educational systems outside the US, like Germany, is less explored. In Germany's Abitur (university entrance exam), teachers grade essays manually, relying on templates that specify expectations for language and content. However, educational science suggests that teachers might be influenced by "construct-irrelevant" factors, such as mechanical accuracy (spelling, grammar), when evaluating essays, potentially impacting grades for unrelated aspects like content. This paper aims to computationally model how task-appropriate language complexity and accuracy affect human grading in this specific German context, bridging computational linguistics and educational science.

Methods

The researchers developed a novel approach to identify "task-appropriate language complexity" (ALC). This involved:

  1. Automatic Complexity Assessment: Extracting a large set of 320 linguistic features from student essays to measure various aspects of language complexity (lexical, phrasal, clausal, discourse, morphological, language use, and processing).
  2. Feature Selection: Using a hybrid approach combining theory-driven (based on educational standards) and data-driven (correlation with overall essay grade) methods to select the most relevant features for each specific task prompt. This resulted in "complexity vectors" for each essay.
  3. Ideal Vector Construction: Creating an "artificial ideal vector" for each task, representing what task-appropriate complexity would look like.
  4. Essay Selection: Ranking essays based on their similarity to this ideal vector. They then selected essays that were either very close to (high ALC) or very far from (low ALC) the ideal, ensuring they were comparable in text length and received a medium overall grade to avoid ceiling/floor effects.
  5. Human Grading Experiment: Having experienced German teachers grade these selected essays for language performance, content, and overall performance, unaware of the essays' ALC status or original grades.
  6. Statistical Modeling: Using linear mixed regression models to analyze how ALC and error rate (a measure of accuracy) influenced the teachers' grades.

Datasets

The study used a dataset of 344 German essays written by students for the 2017 Abitur examination. These essays were collected from four different task prompts (two literature interpretation tasks and two material-based argumentation tasks). No new benchmarks were introduced; the focus was on analyzing existing Abitur data and teacher grading practices.

Key Insights and Ideas

  • Teachers can identify language complexity: The study found that teachers can successfully differentiate essays based on task-appropriate language complexity and use this in their grading for language performance.
  • Accuracy bias is pervasive: A significant finding is that teachers are heavily biased by accuracy (spelling, punctuation, grammar errors) across all grading rubrics, including content and overall performance, even when accuracy is conceptually irrelevant to content evaluation.
  • Disproportionate influence of accuracy: While complexity has a measurable impact on language grades, accuracy's influence is disproportionately strong, particularly on content grades.
  • Task-specific complexity: The study highlights that language complexity features relevant for grading can differ depending on the specific task prompt, suggesting that "task-appropriate" complexity is not a one-size-fits-all concept.
  • Bridging disciplines: The work demonstrates the value of computational linguistics in addressing fundamental questions in educational science, offering a data-driven approach to understanding human judgment in assessment.

Key Results

  • Language Performance Grades: Essays with higher task-appropriate language complexity (+ALC) received approximately 1.37 points higher language performance grades. Error rate, conversely, lowered grades by about 1.99 points.
  • Content Grades: +ALC did not significantly affect content grades. However, error rate lowered content grades by about 1.265 points. This influence was observed for spelling, punctuation, and grammar errors individually.
  • Overall Grades: +ALC marginally increased overall grades by about 0.703 points, while error rate significantly lowered them by about 1.518 points.
  • The models explained a substantial portion of the variance in grades (31.1% to 37.5%), with error rate being a stronger predictor than ALC in most cases.

Limitations and Future Work

  • Ecological Validity vs. Control: The study prioritized ecological validity by using real Abitur essays and experienced teachers. However, this means that essay content and accuracy are intertwined. The authors acknowledge they cannot rule out that students with better spelling also coincidentally produced better content.
  • Future Work:
    • Investigate teacher grading behavior in more depth, potentially clustering teachers based on their characteristics and grading styles.
  • Explore how teachers' personal evaluations of text complexity influence their perception and grading of language quality.
  • Conduct a follow-up study with experimentally manipulated texts (keeping content fixed while varying error rates) to isolate the effect of accuracy more precisely.

Summary updated

BibTeX
@inproceedings{weiss2019computationally,
  author = {Zarah Weiss and Anja Riemenschneider and Pauline Schröter and Detmar Meurers},
  title = {{Computationally Modeling the Impact of Task-Appropriate Language Complexity and Accuracy on Human Grading of German Essays}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure