Get an A in Math: Progressive Rectification Prompting

Zhenyu Wu, Meng Jiang, Chao Shen

Abstract

Chain-of-Thought (CoT) prompting methods have enabled large language models (LLMs) to generate reasoning paths and solve math word problems (MWPs). However, they are sensitive to mistakes in the paths, as any mistake can result in an incorrect answer. We propose a novel method named Progressive Rectification Prompting (PRP) to improve average accuracy on eight MWP datasets from 77.3 to 90.5. Given an initial answer from CoT, PRP iterates a verify-then-rectify process to progressively identify incorrect answers and rectify the reasoning paths. With the most likely correct answer, the LLM predicts a masked numerical value in the question; if the prediction does not match the masked value, the answer is likely incorrect. Then the LLM is prompted to re-generate the reasoning path hinted with a set of incorrect answers to prevent itself from repeating previous mistakes. PRP achieves the best performance compared against the CoT methods.

Introduction

While Chain-of-Thought (CoT) prompting has revolutionized how LLMs handle reasoning, it remains fragile. A single arithmetic slip or a logical "hallucination" early in a reasoning chain cascades into an incorrect final answer. Existing solutions like Self-Consistency (majority voting) can fail if the model has systematic biases, while progressive hinting can anchor the model to incorrect starting points. Furthermore, LLMs are often sensitive to "noise" or irrelevant distractors in math problems.

Progressive Rectification Prompting (PRP) addresses these issues by giving LLMs "exam skills"—specifically the ability to double-check their work and pivot when they realize they’ve made a mistake. By implementing an iterative "verify-then-rectify" loop, PRP boosts average accuracy across eight major benchmarks from 77.3% to 90.5%, outperforming even few-shot methods.

Methods

PRP introduces a two-stage iterative process that requires no manual examples (zero-shot), mimicking human problem-solving techniques.

The Progressive Rectification Prompting (PRP) workflow, showing the iterative verify-then-rectify loop.

1. Substitute Verification (The "Double Check")

Instead of just asking the model "is this right?", PRP forces the model to work backward. The system takes a numerical value from the original question (e.g., "Mary has 5 apples") and replaces it with "X". It then feeds the LLM the masked question along with the model's previously generated answer as a "fact." If the model's answer was correct, it should be able to solve for X and recover the original value (e.g., "5"). If the model calculates X to be something else, it has effectively proven its own previous answer was logically inconsistent.

2. Rectification (The "Pivot")

If verification fails, the model enters the rectification phase. PRP uses a negation hint, re-prompting the model with the instruction: "The answer is likely not [Previous Wrong Answer]." This explicitly tells the LLM to abandon its previous reasoning path and explore a new one. The process iterates (up to a maximum of $K=5$ times) until the verification step confirms the answer is likely correct.

Results and Analysis

The researchers evaluated PRP on eight diverse Math Word Problem (MWP) datasets, including arithmetic-focused (AddSub, SingleOp), algebra-focused (SingleEq), and multi-step reasoning (GSM8K) tasks. They also tested on "noisy" datasets (GSM-IC2-1K and GSM-ICM-1K) containing distractor information.

Performance gains across iterations and average iteration counts for various math word problem datasets.

Key Insights

  • State-of-the-Art Performance: PRP achieved an average accuracy of 90.5% across all datasets, a massive jump from the 77.3% achieved by standard Zero-Shot CoT. It outperformed "Plan-and-Solve" (77.3%) and "Self-Consistency" (70.7% on GSM8K).
  • Robustness to Noise: On the GSM-ICM-1K dataset (full of irrelevant info), PRP improved accuracy by 24.6% compared to standard CoT.
  • Complexity and Iteration: There is a direct correlation between problem difficulty and the number of iterations needed. Simple problems are solved in ~1.5 iterations, while noisy, complex problems require ~4.1 iterations on average.
  • Zero-Shot > Few-Shot: Remarkably, this iterative zero-shot approach outperformed standard few-shot CoT, suggesting that internal logical consistency is more important than following a provided template.

Conclusion

PRP demonstrates that "working backward" is a more robust way for LLMs to identify their own inconsistencies than simple self-checking. By using negation as a tool for exploration, PRP breaks models out of logical ruts. While the iterative nature increases computational cost per problem, the substantial gains in accuracy and robustness to noise make it a powerful tool for complex reasoning tasks. Future work may explore more deliberate masking strategies to improve efficiency and extend this logic to non-mathematical domains.

Summary updated

Abstract

Chain-of-Thought (CoT) prompting methods have enabled large language models (LLMs) to generate reasoning paths and solve math word problems (MWPs). However, they are sensitive to mistakes in the paths, as any mistake can result in an incorrect answer. We propose a novel method named Progressive Rectification Prompting (PRP) to improve average accuracy on eight MWP datasets from 77.3 to 90.5. Given an initial answer from CoT, PRP iterates a verify-then-rectify process to progressively identify incorrect answers and rectify the reasoning paths. With the most likely correct answer, the LLM predicts a masked numerical value in the question; if the prediction does not match the masked value, the answer is likely incorrect. Then the LLM is prompted to re-generate the reasoning path hinted with a set of incorrect answers to prevent itself from repeating previous mistakes. PRP achieves the best performance compared against the CoT methods. Our implementation is made publicly available at https://wzy6642.github.io/prp.github.io/.
BibTeX
@inproceedings{wu2024get,
  author = {Zhenyu Wu and Meng Jiang and Chao Shen},
  title = {{Get an A in Math: Progressive Rectification Prompting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=7w09ofgXHk},
}
Figure