TL;DR
This paper investigates whether automatically scoring only high-confidence answers can save manual grading effort in educational settings. While the approach shows promise, the authors found that confidence thresholds vary significantly across different prompts and datasets, limiting the potential for a universally applicable, cost-effective solution.
What Problem Does This Paper Address?
Manual grading of free-text answers in educational settings is time-consuming and expensive, yet fully automated grading often lacks sufficient accuracy. A common approach to bridge this gap is to use automated systems for answers where the model is highly confident, deferring the rest to human graders. This "confidence-based scoring" aims to reduce manual workload while maintaining high accuracy. However, a key challenge is determining a reliable confidence threshold that works consistently across different questions and contexts, especially in high-stakes exams where re-using questions is limited and the cost of manual effort needs careful consideration.
Methods
The researchers explored three similarity-based scoring methods: edit distance, and two variants of cosine similarity using pre-trained and fine-tuned SBERT (Sentence-BERT) embeddings. For each answer, the model calculates a similarity score to a set of reference answers. This similarity score is then used as a proxy for the model's confidence. The core idea is to set a confidence threshold: if an answer's similarity score exceeds this threshold, it's automatically graded; otherwise, it's sent for manual grading. The study specifically investigates how consistent these confidence thresholds are across different prompts and datasets, a crucial factor for practical implementation.
Datasets
The study utilized five datasets:
- UniversityExams: A newly introduced high-stakes exam dataset of German short answers from a university statistics class. Due to sensitivity, this dataset is not publicly available.
- ASAP: A publicly available dataset of answers to prompts from Biology, Science, and English Language Arts.
- Beetle: A dataset with answers to electricity and electronics questions.
- SEB (SciEntsBank): A dataset containing answers from various science domains.
- Powergrading: A dataset of US Citizenship Exam questions collected via Amazon Mechanical Turk.
The ASAP, Beetle, SEB, and Powergrading datasets were used to provide context for the results obtained on the UniversityExams data.
Key Insights and Ideas
- Confidence is Prompt-Dependent: The most significant finding is that optimal confidence thresholds are not consistent across different prompts within the same dataset. This makes it difficult to establish a single, reliable threshold for a new set of questions.
- High Manual Effort for High Confidence: Even when a suitable threshold can be found for a specific prompt, achieving the target performance often requires a substantial amount of manual scoring, sometimes over half of the answers. This challenges the goal of significantly reducing manual effort.
- Limited Generalizability: The study suggests that there isn't a universal confidence threshold that can be applied across different datasets or prompts to achieve a good balance between performance and manual effort.
- SBERT Shows Slight Advantage: Fine-tuned SBERT embeddings generally offered a slight advantage over edit distance and pre-trained SBERT in fully automated scoring scenarios.
Key Results
The paper reports weighted F1 scores for fully automated scoring, showing that datasets like UniversityExams and Powergrading are generally easier to score automatically. However, when exploring confidence-based scoring, the authors found that the optimal thresholds varied widely for individual prompts. For example, in Figure 2, the red bars (representing the proportion of data below the threshold, i.e., requiring manual scoring) are often substantial, indicating significant manual effort. Figure 3 further illustrates that performance gains from confidence-based scoring often plateau or only increase when manual effort becomes very high, suggesting diminishing returns.
Limitations and Future Work
The authors acknowledge the limitation of not being able to publish the sensitive UniversityExams dataset, which restricts reproducibility. They also note that their calculation of performance excludes answers deferred to manual labeling, which is a practical choice but could be argued differently.
Future work could explore more sophisticated methods for determining confidence thresholds or investigate active learning strategies to iteratively improve models with human feedback on low-confidence answers. The ethical considerations of using automated scoring in high-stakes settings, particularly regarding legal implications and ensuring fairness for all students, are also highlighted as important areas for further discussion.
Summary updated