TL;DR
This paper introduces a novel system that goes beyond simply identifying essay structure to evaluate multiple facets of coherence in student writing. By analyzing semantic similarity and discourse structure, the system aims to provide more nuanced feedback to students, helping them improve their writing quality.
What Problem Does This Paper Address?
While existing automated essay evaluation tools can identify structural elements like thesis statements and main ideas, they often fall short in assessing the quality of those elements and how well they contribute to overall coherence. Teachers frequently observe essays that have all the right structural components but still lack clarity and flow. This paper addresses the need for a system that can evaluate global and local coherence in student essays, providing more granular feedback to writers. Specifically, it tackles the challenge of assessing:
- Relatedness to the essay question (topic): How well does the essay stay on track with the prompt?
- Relatedness between discourse elements: How well do different parts of the essay (e.g., thesis, main ideas, conclusion) connect to each other?
- Intra-sentential quality: How grammatically sound and clear are individual sentences?
- Sentence-relatedness within a discourse segment: How well do sentences within a specific paragraph or section flow together?
Methods
The authors developed a hybrid approach to evaluate coherence. For the global coherence dimensions (relatedness to prompt and relatedness to thesis), they employed a Support Vector Machine (SVM) classifier. This classifier was trained on features derived from:
- Random Indexing (RI): A vector-based method for semantic representation, similar to Latent Semantic Analysis, trained on a large corpus of newswire text. RI features capture semantic similarity between sentences, discourse segments, and the essay prompt.
- Discourse structure information: Features derived from the identified discourse labels of sentences (e.g., thesis, main idea, background).
For the local coherence dimension (intra-sentential quality, termed DimERR), a rule-based heuristic was used. This heuristic automatically assigns high or low ranks based on the presence of grammatical errors, word usage issues, mechanics errors (like spelling or punctuation), incomplete sentences, or run-on sentences.
Datasets
The primary dataset used in this study consists of 989 student essays. These essays were manually annotated by human judges according to a detailed protocol developed by the authors, covering the four coherence dimensions. This annotated corpus serves as the "gold standard" for training and evaluating the developed system. The annotations were built upon existing discourse element labels from the CriterionSM Online Essay Evaluation Service.
Key Insights and Ideas
- Coherence is multi-faceted: The paper strongly advocates for viewing coherence not as a single monolithic concept, but as a combination of global and local aspects, each requiring different evaluation methods.
- Semantic similarity is key for global coherence: Random Indexing proved effective in capturing semantic relationships that are crucial for determining how well sentences relate to the prompt and the essay's thesis.
- Rule-based heuristics are sufficient for local sentence quality: For assessing grammatical correctness and clarity within a sentence, a straightforward rule-based system leveraging existing error detection capabilities is effective.
- The challenge of "low-rank" data: For the dimension of coherence within a discourse segment (Dims), the authors found it difficult to model because most sentences are already highly related to others in their segment, leaving few examples of low coherence for training.
- Human agreement is high: The study highlights that human annotators can achieve high agreement on these coherence dimensions, validating the protocol and the feasibility of automated evaluation.
Key Results
The developed system generally outperforms a simple baseline (which assigns "high" relevance to all sentences) across the evaluated coherence dimensions.
- Dimp (Relatedness to Prompt): The SVM achieved significant improvements over the baseline, particularly for "Background" sentences, suggesting that many students struggle with providing relevant introductory material. The system achieved high recall, meaning it was less likely to incorrectly label a relevant sentence as irrelevant.
- Dimy (Relatedness to Thesis): Similar to Dimp, the SVM showed improvements, though the gains were smaller. The system struggled to outperform the baseline on "Conclusion" sentences, an area needing further investigation.
- DimERR (Technical Errors): The rule-based system significantly outperformed the baseline, demonstrating its effectiveness in identifying sentences with clarity issues.
- Dims (Relatedness within Segment): The paper notes that the baseline performance was already very high (98.1% of sentences were annotated as "highly related"), making it difficult for the system to show substantial improvement.
Limitations and Future Work
- Limited data for intra-segment coherence (Dims): The authors acknowledge that more data needs to be collected and annotated for sentences with low coherence within a discourse segment before this dimension can be effectively modeled.
- Reliance on existing discourse labels: The current system uses human-assigned discourse labels as features. Future work will investigate the impact of using automatically assigned labels from CriterionSM.
- Potential for further refinement: The authors express hope that further investigation of their richly annotated dataset will lead to a more comprehensive writing instruction capability that provides detailed feedback on all coherence dimensions.
Summary updated