Essay Quality Signals as Weak Supervision for Source-based Essay Scoring

Haoran Zhang, Diane LitmanWorkshop 2021

TL;DR

This paper explores using readily available essay features like word count and topic similarity as "weak supervision" to train Automated Essay Scoring (AES) models, aiming to reduce reliance on expensive human-labeled data. While this weak supervision doesn't achieve state-of-the-art performance for direct essay scoring, it proves effective in automatically extracting "Topical Components" (TCs) crucial for a different, feature-based AES system, thus eliminating human effort in both grading and TC creation.

What Problem Does This Paper Address?

Grading student essays is time-consuming and can be inconsistent. Automated Essay Scoring (AES) offers a solution, but most AES systems require large datasets of human-graded essays for training. Creating these datasets is itself a laborious task. This paper tackles the challenge of reducing or eliminating this human effort by investigating "weak supervision" – using automatically derivable signals of essay quality (like word count and topic similarity) instead of human-assigned scores to train AES models. This is particularly relevant for source-based essay tasks, where students must use evidence from a provided text.

Methods

The core of this work lies in exploring two "weak essay quality signals":

  • Word Count (WC): The straightforward count of words in an essay. The intuition is that longer essays might indicate more developed arguments or evidence.
  • Topic Distribution Similarity (TDS): This measures how similar the topics discussed in a student's essay are to the topics present in the source article. This is achieved by training a Latent Dirichlet Allocation (LDA) model on both the student essays and the source text, then comparing their topic distributions using cosine similarity.

These weak signals are used to train a neural network-based AES model (AESneural). The paper then evaluates the effectiveness of these weakly supervised models in two ways:

  1. Direct Essay Scoring (H1): Assessing how well the AESneural model, trained with WC or TDS, predicts human-assigned evidence scores.
  2. Topical Component (TC) Extraction (H2): Investigating if the TCs extracted using weakly supervised methods can be used effectively in a downstream, feature-based AES system (AESrubric). TCs are key concepts and examples from the source text that students are expected to incorporate into their essays.

Datasets

The study uses two corpora for the Response-to-Text Assessment (RTA) task:

  • RTAMVP: 2970 essays based on an article about poverty reduction efforts in Sauri, Kenya.
  • RTASpace: 2076 essays based on an article discussing the importance of space exploration.

These essays were manually graded on a scale of 1 to 4 across five dimensions, with the paper focusing specifically on the "Evidence" dimension.

Key Insights and Ideas

  • Weak Supervision is Not a Silver Bullet for Direct Scoring: While WC and TDS offer some predictive power, they don't reach the performance of human-graded scores when directly training a neural AES model. This highlights the inherent limitations of relying solely on these simpler signals for nuanced essay evaluation.
  • Weak Supervision Shines for TC Extraction: The real win is in using these weak signals to extract Topical Components (TCs). The TCs generated via weakly supervised methods (TCwc and TCtds) perform comparably to those extracted by more complex methods (TCes) or even manually created ones (TCmanual) when used in a downstream feature-based AES system (AESrubric).
  • Automated TCs Can Outperform Manual Ones: Surprisingly, the automated TC extraction methods (especially TCwc and TCtds) sometimes achieved better performance in the downstream AESrubric task than TCs created by human experts. This suggests human subjectivity might introduce noise or suboptimal choices in TC creation.
  • Word Count is a Surprisingly Robust Signal: Despite its simplicity, word count (WC) proved to be a strong performer, often matching or exceeding other automated TC extraction methods, especially when used to train the AESrubric system.

Key Results

  • AESneural Performance (H1): Models trained with human evidence scores significantly outperformed those trained with WC or TDS. However, both WC and TDS significantly outperformed random and majority prediction baselines. TDS generally outperformed WC.
  • TC Extraction Performance (H2a): When using extracted TCs to build features for the AESrubric system, automated methods (TCwc and TCtds) achieved performance comparable to human-created TCs (TCmanual) and a baseline automated method (TCes). TCtds showed particularly strong performance on RTAMVP.
  • TC Feature Correlation (H2b): For the Number of Pieces of Evidence (NPE) feature, TCwc consistently outperformed TCmanual. For the Specificity (SPC) feature, no automated method consistently outperformed TCmanual, but TCwc and TCtds showed competitive results.

Limitations and Future Work

  • Gaming Potential: The authors acknowledge that simple signals like word count could be "gamed" by students writing long, irrelevant essays. However, they argue that since these signals are used for training and not direct prediction, the risk is mitigated, especially if such essays only appear in the testing set.
  • Limited Datasets: The approach was tested on only two RTA corpora. Future work should explore its applicability to other datasets and tasks.
  • Prompt Dependency: Word count's effectiveness might vary if prompts specify a word count. Topic distribution similarity might not be suitable for non-source-based essays.
  • Future Directions: The authors suggest exploring other weak essay quality signals, applying existing signals to different neural architectures and datasets, and investigating the impact of data size. They also aim to make the generated specific examples in TCs more human-readable.

Summary updated

Abstract

Human essay grading is a laborious task that can consume much time and effort. Automated Essay Scoring (AES) has thus been proposed as a fast and effective solution to the problem of grading student writing at scale. However, because AES typically uses supervised machine learning, a human-graded essay corpus is still required to train the AES model. Unfortunately, such a graded corpus often does not exist, so creating a corpus for machine learning can also be a laborious task. This paper presents an investigation of replacing the use of human-labeled essay grades when training an AES system with two automatically available but weaker signals of essay quality: word count and topic distribution similarity. Experiments using two source-based essay scoring (evidence score) corpora show that while weak supervision does not yield a competitive result when training a neural source-based AES model, it can be used to successfully extract Topical Components (TCs) from a source text, which are required by a supervised feature-based AES model. In particular, results show that feature-based AES performance is comparable with either automatically or manually constructed TCs.
BibTeX
@inproceedings{zhang2021essay,
  author = {Haoran Zhang and Diane Litman},
  title = {{Essay Quality Signals as Weak Supervision for Source-based Essay Scoring}},
  booktitle = {Workshop},
  year = {2021},
}
Figure