TL;DR
This paper investigates GPT-3's capabilities in solving math word problems involving systems of linear equations. The authors explore GPT-3's performance on classifying problems, extracting equations, and generating new problems, finding promising results, especially with fine-tuning for equation extraction.
What Problem Does This Paper Address?
Math word problems (MWPs) are a notoriously difficult area for students, requiring them to translate natural language descriptions into mathematical equations. While AI has made strides in solving mathematical problems, accurately extracting the underlying equations from complex word problems remains a significant challenge. This paper aims to evaluate how well GPT-3, a powerful large language model, can handle these tasks, specifically focusing on MWPs that can be represented by systems of two linear equations in two variables. This focus allows for a more controlled study and easier verification of extracted equations.
Methods
The researchers evaluated GPT-3 across three main tasks:
- Classification (Q1): GPT-3 was tasked with categorizing MWPs into five predefined themes (sum and difference, item and property, perimeter of rectangles, motion, and mixture). This leverages GPT-3's natural language understanding for a foundational step in problem-solving.
- Equation Extraction (Q2): This was the primary focus. The team explored various prompting strategies:
- Zero-shot learning: GPT-3 was given a prompt with instructions but no examples.
- Few-shot learning (1-shot to 5-shot): GPT-3 was provided with a varying number of examples in the prompt to guide its output.
-
Fine-tuning: GPT-3 was trained on a larger dataset (L1000) to adapt its parameters specifically for this task. The researchers also investigated the impact of the "temperature" parameter on GPT-3's output, which controls the randomness of its generations.
-
Problem Generation (Q3): GPT-3 was prompted to create new MWPs based on a given example, exploring both within-topic and cross-topic generation.
Datasets
The researchers created several datasets of varying sizes (L20, L30, L50, L100, L200, L1000) by scraping the web for MWPs. These datasets were expanded by programmatically generating new problems with substituted numbers and/or subjects, ensuring no duplicates. The L1000 dataset was specifically created for fine-tuning GPT-3. The datasets were designed to have an equal distribution of the five problem themes.
Key Insights and Ideas
- GPT-3 shows strong potential for MWP tasks: The model demonstrates significant capabilities in classifying, extracting equations from, and generating math word problems.
- Few-shot learning improves equation extraction: The accuracy of extracting equations directly correlates with the number of examples provided in the prompt, with three-shot learning showing a notable improvement over zero-shot.
- Fine-tuning is key for high accuracy: Fine-tuning GPT-3 on a larger dataset significantly boosts equation extraction accuracy, reaching a high of 80%.
- Classification is generally good, but with nuances: GPT-3 performs well across most categories, but struggles with the "item and property" class, often misclassifying them as "mixture."
- Problem generation is effective but topic-dependent: GPT-3 can generate new problems with substituted numbers and subjects, but its success varies by topic, with some categories proving more challenging for cross-topic generation.
- "Sum and Difference" remains a challenge: Even with fine-tuning, GPT-3 struggles to achieve high accuracy on problems categorized as "sum and difference."
Key Results
- Classification: GPT-3 achieved accuracies ranging from 0% to 100% across the five categories. It performed exceptionally well on "mixture" and "perimeter of rectangle" (100%), and "motion" (90%), but struggled with "item and property" (0%).
- Equation Extraction:
- Zero-shot learning: ~31% accuracy.
- Few-shot learning: Accuracy increased from ~31% (zero-shot) to ~69% (three-shot).
- Fine-tuning: Achieved a high accuracy of 80% on the L20 dataset.
- Problem Generation:
- Within-topic generation: Accuracy ranged from 33% to 93% depending on the problem type.
- Cross-topic generation: Showed mixed results, with some categories achieving rates above 70% and others significantly lower.
Limitations and Future Work
- Classification limitations: The "item and property" category proved difficult for GPT-3, suggesting a need for revised or more granular classification schemes. Combining "item and property" and "mixture" into a broader category is suggested as a short-term solution.
- Equation extraction limitations: While fine-tuning achieved 80% accuracy, inconsistencies and random errors in generated equations were still observed. The model sometimes struggled with common-sense reasoning and duplicate problems if it failed on the original.
- Problem generation limitations: The success of generation is topic-dependent, and some topics are too broad. Prompt engineering and better class design are identified as areas for future work.
- Future Work: The authors plan to explore prompts and techniques to guide GPT-3 in explaining how the answers are derived, which is a more challenging task. They also acknowledge that the current study did not cover solving the extracted equations or providing step-by-step instructions, which are important aspects of a complete AI tutor.
Summary updated