TL;DR
This paper explores using readily available essay features like word count and topic similarity as "weak supervision" to train Automated Essay Scoring (AES) models, aiming to reduce reliance on expensive human-labeled data. While this weak supervision doesn't achieve state-of-the-art performance for direct essay scoring, it proves effective in automatically extracting "Topical Components" (TCs) crucial for a different, feature-based AES system, thus eliminating human effort in both grading and TC creation.
What Problem Does This Paper Address?
Grading student essays is time-consuming and can be inconsistent. Automated Essay Scoring (AES) offers a solution, but most AES systems require large datasets of human-graded essays for training. Creating these datasets is itself a laborious task. This paper tackles the challenge of reducing or eliminating this human effort by investigating "weak supervision" – using automatically derivable signals of essay quality (like word count and topic similarity) instead of human-assigned scores to train AES models. This is particularly relevant for source-based essay tasks, where students must use evidence from a provided text.
Methods
The core of this work lies in exploring two "weak essay quality signals":
- Word Count (WC): The straightforward count of words in an essay. The intuition is that longer essays might indicate more developed arguments or evidence.
- Topic Distribution Similarity (TDS): This measures how similar the topics discussed in a student's essay are to the topics present in the source article. This is achieved by training a Latent Dirichlet Allocation (LDA) model on both the student essays and the source text, then comparing their topic distributions using cosine similarity.
These weak signals are used to train a neural network-based AES model (AESneural). The paper then evaluates the effectiveness of these weakly supervised models in two ways:
- Direct Essay Scoring (H1): Assessing how well the AESneural model, trained with WC or TDS, predicts human-assigned evidence scores.
- Topical Component (TC) Extraction (H2): Investigating if the TCs extracted using weakly supervised methods can be used effectively in a downstream, feature-based AES system (AESrubric). TCs are key concepts and examples from the source text that students are expected to incorporate into their essays.
Datasets
The study uses two corpora for the Response-to-Text Assessment (RTA) task:
- RTAMVP: 2970 essays based on an article about poverty reduction efforts in Sauri, Kenya.
- RTASpace: 2076 essays based on an article discussing the importance of space exploration.
These essays were manually graded on a scale of 1 to 4 across five dimensions, with the paper focusing specifically on the "Evidence" dimension.
Key Insights and Ideas
- Weak Supervision is Not a Silver Bullet for Direct Scoring: While WC and TDS offer some predictive power, they don't reach the performance of human-graded scores when directly training a neural AES model. This highlights the inherent limitations of relying solely on these simpler signals for nuanced essay evaluation.
- Weak Supervision Shines for TC Extraction: The real win is in using these weak signals to extract Topical Components (TCs). The TCs generated via weakly supervised methods (TCwc and TCtds) perform comparably to those extracted by more complex methods (TCes) or even manually created ones (TCmanual) when used in a downstream feature-based AES system (AESrubric).
- Automated TCs Can Outperform Manual Ones: Surprisingly, the automated TC extraction methods (especially TCwc and TCtds) sometimes achieved better performance in the downstream AESrubric task than TCs created by human experts. This suggests human subjectivity might introduce noise or suboptimal choices in TC creation.
- Word Count is a Surprisingly Robust Signal: Despite its simplicity, word count (WC) proved to be a strong performer, often matching or exceeding other automated TC extraction methods, especially when used to train the AESrubric system.
Key Results
- AESneural Performance (H1): Models trained with human evidence scores significantly outperformed those trained with WC or TDS. However, both WC and TDS significantly outperformed random and majority prediction baselines. TDS generally outperformed WC.
- TC Extraction Performance (H2a): When using extracted TCs to build features for the AESrubric system, automated methods (TCwc and TCtds) achieved performance comparable to human-created TCs (TCmanual) and a baseline automated method (TCes). TCtds showed particularly strong performance on RTAMVP.
- TC Feature Correlation (H2b): For the Number of Pieces of Evidence (NPE) feature, TCwc consistently outperformed TCmanual. For the Specificity (SPC) feature, no automated method consistently outperformed TCmanual, but TCwc and TCtds showed competitive results.
Limitations and Future Work
- Gaming Potential: The authors acknowledge that simple signals like word count could be "gamed" by students writing long, irrelevant essays. However, they argue that since these signals are used for training and not direct prediction, the risk is mitigated, especially if such essays only appear in the testing set.
- Limited Datasets: The approach was tested on only two RTA corpora. Future work should explore its applicability to other datasets and tasks.
- Prompt Dependency: Word count's effectiveness might vary if prompts specify a word count. Topic distribution similarity might not be suitable for non-source-based essays.
- Future Directions: The authors suggest exploring other weak essay quality signals, applying existing signals to different neural architectures and datasets, and investigating the impact of data size. They also aim to make the generated specific examples in TCs more human-readable.
Summary updated