TL;DR
This paper introduces a novel approach to automatically score open-ended math word problems (MPT) by leveraging structured rubrics to provide explainable predictions. Their method uses a rule-based system that can be trained automatically, demonstrating strong performance on a large dataset and offering insights into student reasoning.
What Problem Does This Paper Address?
Assessing students' understanding of math word problems, especially those requiring explanations beyond just a final answer, is challenging. These "math plus text" (MPT) problems require students to not only compute a value but also articulate their reasoning, often involving a mix of mathematical expressions and natural language. While automated scoring exists for simpler tasks like essays or closed-ended math problems, scoring MPTs is complex due to the varied ways students express mathematical ideas and explanations. Existing systems often struggle to handle the combination of MathML and plain text, and the "partial credit" nature of these problems makes accurate scoring difficult even for human graders. The authors aim to create an automated system that not only scores these problems accurately but also provides clear explanations for its scores, building trust and aiding student learning.
Methods
The core of the proposed solution is a rule-based scoring system designed to be "explainable by construction." This means the scoring logic is directly tied to the problem's rubric. The system uses "rules" that encode mathematical expressions and their equivalent forms (e.g., handling commutativity or float/integer conversions). These rules can also be regular expressions to capture prose-based reasoning.
These rules are then organized into "groups" (sets of rules that capture equivalent mathematical concepts) and "evidence" (combinations of groups that represent a specific aspect of the rubric). Finally, "scorable traits" are formed, mirroring the rubric's components, with positive and negative evidence to account for correct reasoning and misconceptions. A response scores a point for a trait if positive evidence is present and no negative evidence is found.
To overcome the manual effort of defining these rules, the authors explore an automated approach using simulated annealing. This optimization technique iteratively adds, removes, or modifies rules to maximize a performance metric (a combination of accuracy and unweighted average recall). This allows for the discovery of rules and their organization into a scoring model without manual intervention.
Datasets
The authors used a proprietary dataset of 34,417 responses across 14 MPT items. They also adapted an example item from the GSM8K dataset for illustrative purposes. The proprietary dataset covers algebra, arithmetic, and geometry, with scoring scales ranging from 0-2 to 0-4.
Key Insights and Ideas
- Explainability by Design: The rule-based system inherently provides explanations because the scoring logic is directly traceable to the rubric components. This is a significant advantage over "black box" models.
- Handling Mixed Input: The system's normalization process effectively handles responses that combine MathML and plain text, a common challenge in MPTs.
- Automated Rule Discovery: Simulated annealing offers a promising path to scale this explainable scoring approach to a larger number of items, reducing the need for manual rule creation.
- MPT Difficulty: The dataset highlights the inherent difficulty of MPTs for students, with a significant portion of responses receiving a score of 0, underscoring the need for effective assessment tools.
- Transformer Limitations: The paper suggests that off-the-shelf transformer models might not be the most effective approach for MPT scoring, potentially due to the structured and symbolic nature of mathematical reasoning.
Key Results
The manually crafted rule-based model achieved strong performance, comparable to or better than character n-gram random forests and DistilBERT in terms of Unweighted Average Recall (UAR). The automatically discovered rules via simulated annealing performed slightly worse than manually crafted rules but still outperformed the DistilBERT model. While accuracy was similar across some models, UAR was prioritized to better account for performance across all score points, especially rarer ones. The paper shows specific examples where the annealing approach excels (Item 2) and struggles (Item 10), highlighting the nuances of rule discovery.
Limitations and Future Work
The authors acknowledge that their current rule generation process might not capture all sophisticated regular expressions needed for certain items, particularly those involving complex unit conversions or prose-based reasoning. Future work includes improving the candidate rule generation process and exploring how to best convey the generated explanations to stakeholders like students and test administrators. They also plan to investigate the generalizability of their approach to other rubric types and mathematical domains beyond algebra.
Summary updated