TL;DR
This paper introduces Project Essay Grade (PEG), a system that uses statistical and linguistic features (proxes) to automatically grade student essays. The research demonstrates that a computer can achieve a respectable level of accuracy in essay grading, comparable to human experts, by focusing on measurable linguistic features rather than attempting a deep semantic understanding.
What Problem Does This Paper Address?
The paper addresses the challenge of automating the grading of essays, a common but time-consuming task in education. Traditional essay grading relies on human judgment, which can be subjective and inconsistent. The authors explore how computational linguistics and statistical methods can be applied to create an objective and efficient system for evaluating student writing, particularly focusing on the "style" dimension of grading.
Methods
The core methodology involves a two-dimensional approach to essay grading: content and style, and simulation vs. analysis. Project Essay Grade (PEG) primarily focuses on the "simulation" of human judgment for "style" by identifying "proxes" – measurable linguistic features that act as correlates for intrinsic qualities (trins) that human judges value.
The research design follows these steps:
- Human Judgment: Essays are graded by multiple independent human experts to establish a "criterion" (trins).
- Hypothesis Generation: Potential linguistic variables (proxes) are identified that might correlate with human judgments.
- Computer Measurement: Routines are developed to measure these proxes in student essays.
- Data Input: Essays are prepared for computer input, initially through keypunching.
- Output Generation: The computer program processes the essays and generates data on the proxes.
- Statistical Analysis: Multiple regression analysis is used to weight the proxes and maximize the prediction of human ratings.
Datasets
The paper describes the use of essays written by students in Grades 8 to 12 in an American high school.
- Initial Trial: 272 essays were judged by at least four independent teachers.
- Cross-validation: 138 essays were used for cross-validation, comparing the computer's grades with those of five "judges" (four human, one computer).
- Trait Analysis: 256 essays were rated by 32 highly qualified English teachers on five specific traits (ideas/content, organization, style, mechanics, creativity).
Key Insights and Ideas
- Proxies as Effective Correlates: The study demonstrates that measurable linguistic features ("proxes") can effectively simulate human judgment of essay quality, even without deep semantic understanding. This suggests that a significant portion of what humans perceive as good writing can be captured by surface-level linguistic analysis.
- Style is More Amenable to Automation: The research initially focused on stylistic factors, finding them more readily quantifiable and predictable by computer than deeper content analysis.
- Human Judgment Variability: The study highlights the inherent unreliability in human grading, particularly for subjective traits like "creativity." This variability, rather than the computer's limitations, appears to be a bottleneck in achieving perfect prediction.
- Computer Performance Rivals Experts: The computer system achieved a multiple-correlation of .71 with human judgments, a result considered "startling" and comparable to human expert performance.
- "Human-Expert" Job: The computer's ability to predict human ratings was deemed a respectable "human-expert" job.
Key Results
- Overall Accuracy: The initial strategy achieved a multiple-correlation of .71 with human judgments of overall quality for the first set of essays, and by chance, achieved the same coefficient for a second set.
- Predictive Power: Beta weightings derived from one set of essays successfully predicted human judgments for a second set written by the same students.
- Trait-Specific Performance: In the analysis of five specific traits, multiple-regression coefficients (Col. C in Table 3) ranged from .62 (Organization) to .69 (Mechanics), with "Creativity" at .71. After accounting for criterion unreliability (Col. E), these figures improved, with "Creativity" reaching .78.
- Human Reliability: The reliability of human judges for specific traits varied, with "Mechanics" being the most reliably judged (.85) and "Creativity" the least (.72).
Limitations and Future Work
- Focus on Style: The current work primarily addresses the "style" dimension of essay grading, with less emphasis on "subject-matter content" for disciplines like history, philosophy, or science.
- Deep Linguistic Analysis: While the current system is effective, future work aims to incorporate deeper linguistic analysis, including:
- Syntactic Role Assignment (DICT): Expanding dictionaries and ambiguity resolvers.
- Parsing (PARS): Modifying Kuno's program for well-formed substrings.
- Referent Identification (REFER): Identifying pronoun and anaphoric references.
- Propositional Structure (KERNEL and STRUC): Analyzing elementary propositions and their relationships.
- Semantic Equivalence (EQUIV): Augmenting units with semantic information and analyzing redundancy/emphasis.
- Surface Structure Analysis (STYLE): Describing parts of speech, theme organization, sentence structure, etc.
- Content Comparison (CONTNT): Comparing student essays to master essays based on semantic distance.
- Computational Linguistics Challenges: The paper acknowledges that many of the advanced linguistic routines are far from perfected and face "major, stubborn problems in computational linguistics," such as sentence syntactic analysis, the role of semantics in deep structure, and incorporating reader's outside knowledge.
- Practical vs. Ideal: The authors note that for practical essay grading systems, not all advanced routines may be necessary, and approximate solutions can still yield useful results. The goal is to achieve efficient and practicable improvements rather than necessarily perfect humanoid behavior.
Summary updated