Automatic detection of incoherent written responses to open-ended mathematics questions of fourth graders

Felipe Urrutia, Roberto ArayaResearch Square 2023 · DOI

Abstract

Arguing and communicating is one of the basic skills of the mathematics curriculum. Making them in written form facilitates rigorous reasoning. It allows peers to review the argumentation, and to receive feedback from them. Even though they generate additional cognitive efforts to the calculation process, they enhance long-term retention and facilitate deeper understanding. However, developing arguing and communicating competences in elementary school classrooms are a great challenge. It requires at least two conditions: all students write and all receive immediate feedback. One solution is to use online platforms. However, this is very demanding for the teacher. The teacher must review 30 answers in real-time. To facilitate the revision, it is necessary to automatize the detection of incoherent responses. Thus, the teacher can immediately demand to correct them. Some incoherent answers are obvious. Others are more complex. They require understanding and comparison with the question. In this work, we analyze 14,457 responses to open-ended questions written by 974 fourth graders on the ConectaIdeas online platform. 13% of the answers are incoherent. Using natural language processing and machine learning algorithms, we build an automatic classifier. Then, we tested the classifier on an independent set of written responses to different open-ended questions. We found that the classifier achieves F1-score = 79.15% for incoherent detection, which is better than baselines using different heuristics.

Introduction

In modern mathematics education, the ability to "argue and communicate" reasoning is a fundamental skill. Writing justifications for mathematical solutions encourages rigorous thinking and deeper conceptual understanding. However, implementing this in a classroom setting presents a significant logistical challenge for teachers. In a typical session using online platforms like ConectaIdeas, a teacher may receive dozens of open-ended responses simultaneously. To provide the immediate feedback necessary for effective learning, the teacher has very little time to review each submission.

A major obstacle is the prevalence of incoherent responses—answers that are gibberish, off-topic, or logically disconnected from the prompt (e.g., answering "no" to a calculation request). This study aims to automate the detection of these "low-effort" responses using Natural Language Processing (NLP) and machine learning. By filtering out incoherence, the system allows teachers to focus their human attention on students who have provided coherent but perhaps mathematically incorrect reasoning.

Methodology

The researchers analyzed a dataset of 14,457 responses from 974 fourth graders across 716 different questions. They explored three distinct families of models to classify responses as "coherent" or "incoherent":

  1. Shallow Models: Traditional machine learning techniques like SVM and XGBoost, utilizing hand-engineered features such as word count, punctuation, and part-of-speech tags.
  2. Deep Models: Leveraging BETO (a Spanish version of BERT), the team tested standard fine-tuning against a more sophisticated multi-task learning strategy. In the multi-task approach, the model was trained to simultaneously predict both the incoherence of an answer and the specific type of question asked (e.g., "Calculate with explaining" vs. "Compare quantities").
  3. Ensemble Models: A general architecture that used an XGBoost classifier to aggregate the probability outputs from various specialized sub-classifiers.
Overview of the experimental workflow, including data preparation, model architectures, and evaluation metrics.

The study utilized two primary datasets from the ConectaIdeas platform: a 2019 dataset for training and validation, and an independent 2017 dataset for testing generalization across different school years and student cohorts.

Key Results

The BETO Multi-tasking model emerged as the top performer, achieving an F1-score of 79.15% on the independent test set. This approach demonstrated that forcing the model to identify the nature of the math problem (the "question type") significantly improved its ability to judge the coherence of an answer within that specific context.

Key findings include:

  • Contextual Sensitivity: The model successfully distinguished between question-independent incoherence (gibberish) and question-dependent incoherence where the answer might be a valid word but makes no sense for the specific prompt.
  • Robustness: Deep learning models proved more resilient than shallow models when transferred to the independent 2017 dataset, suggesting they captured deeper linguistic patterns rather than simple keyword associations.
  • Precision vs. Recall: The General Ensemble model achieved the highest precision (78.79%), which is vital in a classroom to minimize "false alarms" that might frustrate students who are genuinely trying.

Conclusion and Future Work

Automating the detection of incoherent responses provides a scalable solution for managing large volumes of student data in real-time. While the current system is highly effective, the authors suggest several avenues for improvement. The development of a "MathBERT"—specifically pre-trained on elementary mathematical vocabulary—could further enhance accuracy. Additionally, incorporating eXplainable AI (XAI) features would help teachers understand the reasoning behind the model's flags, further bridging the gap between automated tools and pedagogical needs in the classroom.

Summary updated

Abstract

<title>Abstract</title> Arguing and communicating is one of the basic skills of the mathematics curriculum. Making them in written form facilitates rigorous reasoning. It allows peers to review the argumentation, and to receive feedback from them. Even though they generate additional cognitive efforts to the calculation process, they enhance long-term retention and facilitate deeper understanding. However, developing arguing and communicating competences in elementary school classrooms are a great challenge. It requires at least two conditions: all students write and all receive immediate feedback. One solution is to use online platforms. However, this is very demanding for the teacher. The teacher must review 30 answers in real-time. To facilitate the revision, it is necessary to automatize the detection of incoherent responses. Thus, the teacher can immediately demand to correct them. Some incoherent answers are obvious. Others are more complex. They require understanding and comparison with the question. In this work, we analyze 14,457 responses to open-ended questions written by 974 fourth graders on the ConectaIdeas online platform. 13\% of the answers are incoherent. Using natural language processing and machine learning algorithms, we build an automatic classifier. Then, we tested the classifier on an independent set of written responses to different open-ended questions. We found that the classifier achieves F1-score = 79.15\% for incoherent detection, which is better than baselines using different heuristics.
BibTeX
@inproceedings{urrutia2023automatic,
  author = {Felipe Urrutia and Roberto Araya},
  title = {{Automatic detection of incoherent written responses to open-ended mathematics questions of fourth graders}},
  booktitle = {Research Square},
  year = {2023},
  doi = {10.21203/rs.3.rs-2566472/v1},
}
Figure