TL;DR
This paper presents a prototype system for automatically scoring Advanced Placement (AP) Biology essays using natural language processing (NLP) techniques. By developing an example-based lexicon and concept grammars, the system achieves high agreement with human raters, demonstrating the potential for efficient and consistent automated assessment in specialized scientific domains.
What Problem Does This Paper Address?
Standardized tests often rely on multiple-choice questions, which may not fully capture a student's understanding of complex scientific concepts. Constructed-response questions, like essays, offer a richer assessment but are time-consuming and costly to score manually. This paper addresses the challenge of automating the scoring of AP Biology essays, aiming to provide an efficient, consistent, and potentially diagnostic tool for evaluating students' scientific knowledge as expressed in natural language. The goal is to bridge the gap between the depth of essay assessment and the practicalities of large-scale standardized testing.
Methods
The core of the system relies on an example-based approach to NLP. Key methods include:
- Lexicon Development: A lexicon was built using an example-based approach, compiling words and terms from 100 training essays that represented the core meaning of sentences. This involved identifying metonyms (acceptable substitutions for words) to capture semantic similarity, allowing the system to recognize paraphrased information.
- Concept-Structure Representations (CSRs): To handle variations in sentence structure and wording, the system parses sentences using Microsoft's Natural Language Processing Tools (MSNLP). It then substitutes words with superordinate concepts from the lexicon and extracts phrasal nodes, creating CSRs that represent the conceptual content and phrasal structure of a sentence. This approach prioritizes structural patterns over simple string matching for semantic analysis.
- Computer Rubric: A computer-based rubric was manually created, mirroring the categories in the human scoring guide. Each category contains concept grammar rules derived from the CSRs of relevant sentences in the training data.
- Automatic Rule Generation: A program automatically generates concept grammar rules by creating permutations of the fine-tuned CSRs. This significantly speeds up the process compared to manual rule creation.
- Automated Scoring: The scoring program uses a searching algorithm to find matches between CSRs (or subsets of CSRs) and the generated concept grammar rules. Points are assigned based on these matches, and an overall essay score is calculated. Essays scoring 9 or more points are classified as "Excellent."
Datasets
- Training Data: 100 "Excellent" AP Biology essays were used for training. From these, 85 were ultimately usable for the study.
- Test Data: 105 test essays were used for evaluation, comprising the remaining 85 "Excellent" essays and 20 "Poor" essays. The "Poor" classification was defined for the study as essays scoring 0-3 points, distinct from the official AP classification.
Key Insights and Ideas
- The Power of Example-Based Learning for Domain-Specific Lexicons: The study highlights how using actual examples from the target domain (AP Biology essays) is crucial for building a lexicon that captures the nuanced vocabulary and semantic relationships relevant to scientific concepts.
- CSRs as a Robust Representation for Semantic Equivalence: By focusing on conceptual content and phrasal structure rather than exact wording, CSRs enable the system to recognize paraphrased ideas, a critical feature for scoring natural language responses.
- Automated Rule Generation as a Significant Efficiency Gain: The ability to automatically generate concept grammar rules from CSRs drastically reduces the time and effort required for system development, making automated scoring more scalable.
- Defensibility of the Scoring Approach: The system's scores are directly linked to the analysis of lexical meanings and syntactic structures, which are then compared against a human-expert-developed scoring guide, making the scoring process transparent and defensible.
Key Results
The prototype scoring system achieved:
- 94% agreement on exact or adjacent scores between computer-based scores and human rater scores for 105 test essays.
- 87% exact agreement with human grader scores for the same set of essays.
- Coverage (Cov) of 100% for both "Excellent" and "Poor" datasets, meaning all essays were assigned a score.
- Accuracy (Acc) of 89% for "Excellent" essays and 75% for "Poor" essays.
- Accuracy within 1 point (Acc w/i 1) of 95% for "Excellent" and 90% for "Poor" essays.
- Accuracy within 2 points (Acc w/i 2) of 100% for "Excellent" and 95% for "Poor" essays.
Limitations and Future Work
- Lexicon Deficiency: The study acknowledges that the lexicon built from the training data may not cover all vocabulary present in the test data, leading to incomplete CSRs and degraded scoring. Future work suggests expanding the lexicon using standard dictionary sources and corpus-based techniques from domain-specific texts (e.g., Biology textbooks).
- Concept Grammar Rule Deficiency: Cases where essay information is expressed in novel ways not covered by existing rules can lead to lower scores. Future research could explore methods to automatically generate rules for these novel expressions.
- Automation of CSR Fine-Tuning: The process of removing extraneous concepts from CSRs is currently manual. Further research is needed to automate this step for greater efficiency.
- Discourse-Level Analysis: The current system primarily focuses on sentence-level analysis. Addressing more sophisticated responses that involve discourse-level understanding will require further development.
- Broader Applications: The methods are seen as applicable to various testing, training, and information retrieval tasks, particularly for scientific or technical texts.
Summary updated