Improving Mathematics Tutoring With A Code Scratchpad

Shriyash Upadhyay, Etan Ginsberg, Chris Callison-BurchACL 2023

TL;DR

This paper explores using Large Language Models (LLMs) as intelligent tutoring systems (ITS) for elementary math. They found that providing LLMs with a "code scratchpad" significantly improves their ability to act as tutors, leading to higher accuracy in explaining and correcting student errors compared to text-only scratchpads.

What Problem Does This Paper Address?

Creating effective Intelligent Tutoring Systems (ITS) is a notoriously difficult and time-consuming process, often relying on complex rule-based systems. While LLMs have shown promise in solving math problems through rationale generation and in improving dialogue systems, their application as full-fledged tutors – capable of not just solving problems but also explaining solutions and guiding students – has been less explored. This paper investigates whether LLMs can effectively perform the core tasks of an ITS, specifically focusing on mathematics at an elementary school level, and whether a code-based "scratchpad" can enhance their tutoring capabilities.

Methods

The authors frame the problem of ITS as a mapping of traditional ITS components (domain model, student model, tutoring model, user interface) to LLM capabilities. They evaluate LLMs on three key sub-tasks:

  1. Math Problem Solving: Generating correct numerical answers.
  2. Explanation Generation: Providing clear and complete explanations for the solutions.
  3. Correction Generation: Identifying and explaining errors in student-provided answers.

The core novelty lies in comparing two LLM configurations: one using a standard text-based scratchpad for generating rationales and another using a code-based scratchpad (Python) where the code is executed to derive the answer. For code-based explanations, the model converts the generated code into natural language. Human annotators evaluate the quality of explanations and corrections based on criteria like validity, completeness, correctness, relevance, and interpretability.

Datasets

The study uses the SVAMP dataset, a collection of 1000 arithmetic word problems designed to challenge existing math problem-solving models. The dataset includes problem narratives, questions, and correct numerical solutions.

Key Insights and Ideas

  • Code Scratchpads are Key: The most significant finding is that LLMs equipped with a code scratchpad dramatically outperform those using only text scratchpads for tutoring tasks. This suggests that the structured, executable nature of code provides a more robust "thinking space" for the LLM.
  • Beyond Just Answers: While LLMs are good at generating answers, the real challenge for tutoring lies in explanation and correction. Code scratchpads show a marked improvement in the correctness and completeness of these tutoring functions.
  • Code to Text Conversion is Tricky: While code scratchpads improve overall tutoring, converting the generated code into natural language explanations can introduce errors. Text-based scratchpads, by generating "chain-of-thought" directly, sometimes produce more correct explanations, but often at the cost of completeness.
  • Human Evaluation is Crucial: The paper emphasizes the need for human evaluation to assess the nuanced quality of explanations and corrections, as simple accuracy metrics are insufficient for evaluating tutoring effectiveness.

Key Results

The paper reports a significant improvement in cumulative accuracy for GPT-3 models using code scratchpads compared to text scratchpads.

  • Cumulative Accuracy: 77.7% (code scratchpad) vs. 48.7% (text scratchpad).
  • Sub-task Performance (Code vs. Text):
    • Solved: 79.4% vs. 63.7%
    • Explained: 98.9% vs. 97.9%
    • Corrected: 99.0% vs. 78.1%

While code scratchpads excel in generating corrections, explanations generated by text scratchpads were found to be more likely to be correct, though less complete.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study focuses solely on elementary mathematics and doesn't generalize to other subjects or more complex math.
  • LLM Specificity: The comparison is limited to two specific GPT-3 models (text-davinci-002 and code-davinci-002).
  • Necessary vs. Sufficient Conditions: The paper tests the ability of LLMs to perform ITS functions but doesn't claim these are sufficient conditions for a fully effective tutor (e.g., engaging interaction, direct question answering).
  • Manual Effort: The manual effort required for evaluation limited the number of comparisons and the scale of the experiments.

Future work should explore:

  • Broader Applicability: Investigating LLMs for tutoring in other domains and at higher educational levels.
  • New Evaluation Methods: Developing more sophisticated ways to assess LLM tutoring performance.
  • Error Reduction: Focusing on techniques to further minimize errors in LLM-generated explanations and corrections.
  • Model Distillation: Exploring smaller, more cost-effective models for ITS deployment.

Summary updated

BibTeX
@inproceedings{upadhyay2023improving,
  author = {Shriyash Upadhyay and Etan Ginsberg and Chris Callison-Burch},
  title = {{Improving Mathematics Tutoring With A Code Scratchpad}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure