A Reliable Approach to Automatic Assessment of Short Answer Free Responses

Lyle F. Bachman, Nathan Carr, Greg Kamei, Mikyung Kim, Michael J. Pan, Chris Salvador, Yasuyo SawakiCOLING 2002

TL;DR

This paper introduces WebLAS, a web-based system for automatically scoring short, free-response language answers. It tackles the inefficiencies of manual grading by using natural language processing to create and apply scoring rubrics, offering a more reliable and explainable alternative to existing automated systems.

What Problem Does This Paper Address?

Traditional paper-and-pencil language assessments are administratively costly and inefficient, especially for free-response questions. While some automated scoring exists, current systems often focus on holistic essay scoring (like e-Rater) or superficial linguistic analysis (like PEG), or require massive computational resources and are opaque to educators (like MITRE and ATM). There's a clear need for a system that can reliably and explainably score short, limited-production free responses in an educational context, allowing for partial credit and transparency.

Methods

WebLAS employs a novel approach to automatic scoring by mimicking the human process of rubric creation. The system guides instructors through a task creation module that involves:

  • Interactive Rubric Generation: Instructors provide a "gold standard" model answer. WebLAS uses its Natural Language Processing (NLP) modules, specifically the Link Grammar Parser and WordNet lexicon, to parse the model answer, identify key elements (phrases or single words), and suggest alternatives.
  • Instructor Confirmation: Instructors confirm these elements and alternatives, assigning point values. This interactive process allows for partial credit scoring.
  • Regex-based Scoring Key: WebLAS translates these confirmed elements, alternatives, and score assignments into a scoring key using regular expressions (regex). This key then matches patterns in student responses.
  • Computational Efficiency: By preprocessing the search for scoring elements into regexes, WebLAS achieves a constant time complexity (O(1)) for scoring each student response, significantly reducing redundancy compared to systems that perform searches anew for each response.

Datasets

Not discussed in the paper. The system is designed to work with instructor-provided model answers and uses existing linguistic resources like WordNet and the Link Grammar Parser.

Key Insights and Ideas

  • Mimicking Human Expertise: The core innovation is simulating the human process of rubric creation and application, making the system more intuitive for educators and the scoring more explainable.
  • Partial Credit for Nuance: By allowing instructors to define elements and assign partial credit, WebLAS moves beyond simple right/wrong scoring, better reflecting the complexity of language production.
  • Explainable AI for Education: The system's reliance on instructor confirmation and regex-based scoring makes the scoring decisions transparent and justifiable to both instructors and students, addressing a major drawback of many black-box automated systems.
  • Computational Efficiency through Preprocessing: The use of regexes generated during task creation significantly speeds up the scoring process, making it scalable.

Key Results

The paper focuses on the methodology and design of WebLAS, rather than presenting quantitative results from a large-scale evaluation. The key results are qualitative and conceptual:

  • Reliability and Robustness: The approach is presented as promising due to its potential for reliability and robustness, stemming from the structured rubric creation and repeatable scoring process.
  • Intra-rater Reliability: The system boasts perfect intra-rater reliability because scores are repeatable and do not change with each scoring instance.
  • Explainable and Justifiable Scores: Scores are explainable and justifiable because instructors confirm all scoring decisions beforehand.

Limitations and Future Work

  • Not a Learning System (Yet): The current system does not learn from student responses; it relies on pre-defined regexes.
  • Limited NLP Capabilities: Future work aims to incorporate more advanced NLP algorithms, such as inference and pronoun resolution, to handle more complex response types.
  • Expansion of Task Types: The authors plan to include additional task types like summarization, outlining, and gap-fill tasks.
  • Online Scoring and Instant Feedback: The immediate goal is to bring the scoring online and provide students with instantaneous feedback.
  • Pilot Testing: Pilot testing was scheduled for Winter and Spring 2003, with a full campus rollout planned for Fall 2003.

Summary updated

BibTeX
@inproceedings{bachman2002reliable,
  author = {Lyle F. Bachman and Nathan Carr and Greg Kamei and Mikyung Kim and Michael J. Pan and Chris Salvador and Yasuyo Sawaki},
  title = {{A Reliable Approach to Automatic Assessment of Short Answer Free Responses}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2002},
}
Figure