Abstract
Large Language Models (LLMs) have demonstrated significant potential in solving mathematical word problems (MWPs) through Chain-of-Thought (CoT) prompting. However, static few-shot prompts often struggle with generalization across diverse problem types and lack a rigorous mechanism for verification. This paper introduces a framework that transforms standard training data into executable Python programs to serve as intermediate reasoning steps. By implementing Dynamic Program Prompting to retrieve contextually relevant exemplars and Program Distillation to train smaller models, the authors achieve state-of-the-art performance on major benchmarks, including GSM8K, SVAMP, and MathQA.
Introduction
While CoT prompting allows models to decompose problems into step-by-step natural language explanations, it faces two primary hurdles: the "fuzziness" of natural language makes automatic verification difficult, and a fixed set of prompts rarely covers the breadth of real-world mathematical variety. This research explores moving beyond static prompts by treating reasoning as code execution, which provides an explicit and verifiable logic path.
Methodology
The proposed approach utilizes a multi-stage pipeline to enhance numerical reasoning capabilities:
- Automatic Program Annotation: The researchers use a high-capacity model (Codex) to generate Python programs for existing training instances. These programs are executed, and only those whose output matches the ground-truth answer are retained as "pseudo-gold" reasoning paths.
- Dynamic Program Prompting: Rather than using a fixed set of examples, the system employs sentence embeddings (e.g., Sentence-T5) to retrieve the $M$ most similar problems from the annotated training set for each new test query. This ensures the model receives a relevant "template" for the specific problem type.
- Program Distillation: To bridge the gap between massive LLMs and smaller models, a 6-billion parameter model (CodeGen) is fine-tuned on the verified programs, allowing it to inherit the rigorous reasoning structure of larger models.
Experimental Results
The framework was evaluated against several competitive baselines across three standard benchmarks: GSM8K (grade school math), SVAMP (robustness challenge), and MathQA (diverse subjects like physics and geometry).
Performance Gains
As shown in the results table below, Dynamic Program Prompting significantly outperforms existing methods. On the GSM8K dataset, the 175B parameter model achieved 76.6%, surpassing the previous PAL (Program-Aided Language model) baseline. The most dramatic improvement occurred on the complex MathQA dataset, where performance jumped from approximately 30% to 61.7%.
Efficiency through Distillation
The distillation experiments revealed that code-based reasoning is a highly effective target for smaller neural networks. The 6B parameter model trained via Program Distillation achieved 39.0% on GSM8K, outperforming an 11B parameter model fine-tuned on natural language reasoning chains (38.2%).
Discussion and Conclusion
The findings underscore that code is a more rigorous medium for reasoning than natural language. In a CoT setting, a model might arrive at a correct answer through flawed logic; however, a program that executes correctly is far more likely to represent sound reasoning. Furthermore, the success of dynamic retrieval suggests that for complex mathematical tasks, the proximity of the few-shot exemplars to the target problem is a critical factor in model success.
Future work will address the computational overhead associated with dynamic retrieval and explore ways to further close the performance gap between distilled small models and their 175B parameter counterparts.
Summary updated