Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning

Zhanming Jie, Wei Lu

Abstract

Large Language Models (LLMs) have demonstrated significant potential in solving mathematical word problems (MWPs) through Chain-of-Thought (CoT) prompting. However, static few-shot prompts often struggle with generalization across diverse problem types and lack a rigorous mechanism for verification. This paper introduces a framework that transforms standard training data into executable Python programs to serve as intermediate reasoning steps. By implementing Dynamic Program Prompting to retrieve contextually relevant exemplars and Program Distillation to train smaller models, the authors achieve state-of-the-art performance on major benchmarks, including GSM8K, SVAMP, and MathQA.

Introduction

While CoT prompting allows models to decompose problems into step-by-step natural language explanations, it faces two primary hurdles: the "fuzziness" of natural language makes automatic verification difficult, and a fixed set of prompts rarely covers the breadth of real-world mathematical variety. This research explores moving beyond static prompts by treating reasoning as code execution, which provides an explicit and verifiable logic path.

Methodology

The proposed approach utilizes a multi-stage pipeline to enhance numerical reasoning capabilities:

  1. Automatic Program Annotation: The researchers use a high-capacity model (Codex) to generate Python programs for existing training instances. These programs are executed, and only those whose output matches the ground-truth answer are retained as "pseudo-gold" reasoning paths.
  2. Dynamic Program Prompting: Rather than using a fixed set of examples, the system employs sentence embeddings (e.g., Sentence-T5) to retrieve the $M$ most similar problems from the annotated training set for each new test query. This ensures the model receives a relevant "template" for the specific problem type.
  3. Program Distillation: To bridge the gap between massive LLMs and smaller models, a 6-billion parameter model (CodeGen) is fine-tuned on the verified programs, allowing it to inherit the rigorous reasoning structure of larger models.
Figure 1: Overview of the proposed framework, contrasting traditional CoT with Program Prompting and the iterative annotation loop.

Experimental Results

The framework was evaluated against several competitive baselines across three standard benchmarks: GSM8K (grade school math), SVAMP (robustness challenge), and MathQA (diverse subjects like physics and geometry).

Performance Gains

As shown in the results table below, Dynamic Program Prompting significantly outperforms existing methods. On the GSM8K dataset, the 175B parameter model achieved 76.6%, surpassing the previous PAL (Program-Aided Language model) baseline. The most dramatic improvement occurred on the complex MathQA dataset, where performance jumped from approximately 30% to 61.7%.

Table 2: Comparison of Dynamic Program Prompting and Program Distillation against state-of-the-art prompting and fine-tuning baselines.

Efficiency through Distillation

The distillation experiments revealed that code-based reasoning is a highly effective target for smaller neural networks. The 6B parameter model trained via Program Distillation achieved 39.0% on GSM8K, outperforming an 11B parameter model fine-tuned on natural language reasoning chains (38.2%).

Discussion and Conclusion

The findings underscore that code is a more rigorous medium for reasoning than natural language. In a CoT setting, a model might arrive at a correct answer through flawed logic; however, a program that executes correctly is far more likely to represent sound reasoning. Furthermore, the success of dynamic retrieval suggests that for complex mathematical tasks, the proximity of the few-shot exemplars to the target problem is a critical factor in model success.

Future work will address the computational overhead associated with dynamic retrieval and explore ways to further close the performance gap between distilled small models and their 175B parameter counterparts.

Summary updated

Abstract

Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging, especially in the context of math word problem (MWP) solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our approach is largely inspired by Gao et al., (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving. Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the program-annotated training data. Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning. Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of fine-tuned small models in MWP solving.
BibTeX
@inproceedings{jie2023leveraging,
  author = {Zhanming Jie and Wei Lu},
  title = {{Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=xV2W4TpfO8s},
}
Figure