Abstract
Solving multi-step math word problems (MWPs) requires complex reasoning and the ability to map natural language to mathematical operations. While Large Language Models (LLMs) have shown promise in this area, they often struggle with long-haul logic. This paper explores a pedagogical approach known as Socratic scaffolding—breaking down complex problems into focused, goal-driven sub-questions. We present a system developed by researchers at ETH Zurich that utilizes a two-pronged approach involving content planning and reinforcement learning to generate these questions automatically. Our results demonstrate that these generated sub-questions significantly improve the performance of AI solvers and provide a framework for digital educational tools.
Introduction
In the realm of mathematics education, simply providing a student with a final answer is often less effective than guiding them through the reasoning process. This method, known as Socratic questioning, involves asking leading questions that help the learner discover the solution themselves. However, automating this process for AI is difficult; existing models typically focus on single-hop factual questions rather than the sequential, logic-heavy inquiries required for math.
This research addresses the challenge of generating sequences of sub-questions that are both mathematically sound and educationally useful. By focusing on "scaffolding," the system aims to bridge the gap between a problem statement and its final solution, helping both machines and humans navigate complex logical paths.
Methodology
The proposed system ensures that generated questions are focused on critical points and lead toward the final answer through two primary components:
1. Content Planning (The "Focus")
To prevent the model from drifting off-track, the system employs a "Content Planner" based on a Seq2Seq architecture. Before any questions are generated, this planner extracts a mathematical plan from the problem text. This plan identifies the specific operators (e.g., addition, multiplication) or equations necessary to solve the problem. This structured plan acts as a constraint, ensuring the subsequent questions remain grounded in the required mathematical logic.
2. Reinforcement Learning (The "Goal")
The Question Generator, built on the T5 architecture, is fine-tuned using the REINFORCE algorithm. The model is optimized based on three distinct reward metrics:
- Fluency: Ensures the questions are grammatically correct and readable.
- Granularity: Matches the number of sub-questions to the number of logical steps required by the problem.
- Answerability: This novel metric uses a separate "student" model (GPT-2) to test the effectiveness of the questions. If the student model can successfully reach the correct final answer using the generated sub-questions, the generator receives a higher reward.
Experimental Setup and Datasets
The study primarily utilizes the GSM8K (Grade School Math 8K) dataset, a benchmark of 8,500 high-quality, multi-step grade school math word problems. GSM8K is specifically designed to test multi-step reasoning rather than simple arithmetic, making it an ideal environment for testing Socratic scaffolding.
Key Results
The implementation of Socratic sub-questions yielded significant improvements in AI reasoning performance:
- AI Performance Boost: When provided with generated sub-questions, a GPT-2 solver's accuracy jumped from 5.45% to 10.46%. More impressively, a one-shot GPT-3 model saw accuracy rise from 29% to 47%.
- Question Quality: Integrating equations into the content planner improved the BLEU score (measuring text similarity to ground truth) from 51.53 to 58.82, indicating more accurate and relevant question sequences.
- Human Evaluation: In trials with human users, the model's questions were rated highly for logical relevance and sequential integrity, outperforming baseline models that lacked content planning or RL-based rewards.
Discussion and Insights
Several critical insights emerged from the study regarding the nature of automated tutoring:
Sequential Integrity: The researchers discovered that the order of questions is as important as the content. When the sequence of sub-questions was permuted, the performance of math solvers dropped significantly, confirming that Socratic tutoring relies on a specific logical progression.
AI as a Proxy for Humans: Using a smaller LLM (GPT-2) as a "proxy student" during training allowed the "teacher" model (T5) to learn how to bridge specific reasoning gaps. This suggests that smaller models can effectively serve as training environments for more complex pedagogical agents.
The Difficulty Threshold: A human user study revealed a "helpfulness gap." While sub-questions significantly aided users with simpler problems, they occasionally increased the "cognitive load" on very complex problems. If the generated questions were not near-perfect, they risked confusing the learner rather than assisting them.
Conclusion
This research demonstrates that LLMs can be trained to provide effective mathematical scaffolding through automated Socratic questioning. While the current system is static—generating a fixed sequence of questions—it lays the groundwork for future dialogic tutoring systems. Such systems would be adaptive, adjusting their questions in real-time based on specific student misconceptions. To reach this goal, future work must focus on increasing the granularity and factuality of questions to ensure they remain helpful even as problem complexity increases.
Summary updated