Abstract
Chain-of-Thought (CoT) prompting methods have enabled large language models (LLMs) to generate reasoning paths and solve math word problems (MWPs). However, they are sensitive to mistakes in the paths, as any mistake can result in an incorrect answer. We propose a novel method named Progressive Rectification Prompting (PRP) to improve average accuracy on eight MWP datasets from 77.3 to 90.5. Given an initial answer from CoT, PRP iterates a verify-then-rectify process to progressively identify incorrect answers and rectify the reasoning paths. With the most likely correct answer, the LLM predicts a masked numerical value in the question; if the prediction does not match the masked value, the answer is likely incorrect. Then the LLM is prompted to re-generate the reasoning path hinted with a set of incorrect answers to prevent itself from repeating previous mistakes. PRP achieves the best performance compared against the CoT methods.
Introduction
While Chain-of-Thought (CoT) prompting has revolutionized how LLMs handle reasoning, it remains fragile. A single minor error in an intermediate step usually leads to a completely wrong final answer. Existing solutions like Self-Consistency (majority voting) often fail because models tend to repeat systematic mistakes. Furthermore, most models lack a mechanism to "double-check" their work—by working the problem backward or checking if the answer is logically consistent with the inputs.
Progressive Rectification Prompting (PRP) addresses these gaps by giving LLMs "exam skills" to solve complex math word problems. By iteratively verifying answers through a "substitute-and-solve" approach and providing negative feedback on incorrect attempts, PRP boosts average accuracy across major benchmarks to an "A-level" 90.5%.
Methodology
PRP mimics the human "dual process theory" of cognition: an intuitive first pass followed by deliberate, reflective refinement. The process involves three main stages:
- Initialization: The LLM generates an initial answer using standard Zero-Shot CoT ("Let's think step by step").
- Verification Module (Substitute Verification): Instead of asking the model "Is this answer right?", PRP masks a numerical value in the original question (turning it into $X$) and treats the generated answer as a known fact. It then asks the LLM to solve for $X$. If the model's predicted $X$ doesn't match the original value from the question, the answer is flagged as incorrect.
- Rectification Module (Negation Hints): If an answer is flagged, it is added to a "potentially incorrect" list. The LLM is then prompted to try again with a specific hint: "The answer is likely not [Previous Answers]." This forces the model to abandon its previous flawed reasoning path and explore new ones.
This "verify-then-rectify" loop continues until the verification passes or a maximum number of iterations (usually 5) is reached.
Experimental Results
The researchers evaluated PRP on eight diverse Math Word Problem (MWP) datasets, including standard benchmarks (AddSub, MultiArith, SVAMP, GSM8K) and challenging "Irrelevant Context" benchmarks (GSM-IC2-1K and GSM-ICM-1K) that contain distractor information.
Using text-davinci-003 as the backbone, PRP achieved state-of-the-art results:
- Average Accuracy: Jumped to 90.5%, outperforming the previous best zero-shot method (77.3%) and surpassing the best few-shot methods (81.0%).
- Complex Reasoning: On the difficult GSM-IC2-1K dataset, PRP reached 93.1% accuracy, compared to just 70.7% for standard CoT.
- Robustness to Noise: In the GSM-ICM-1K dataset, which is full of distractors, PRP improved accuracy by nearly 25% compared to standard Zero-Shot CoT.
Key Insights
- The Power of Negation: Telling a model what the answer isn't is often more helpful than giving it a vague hint about what the answer is. It prevents the model from getting stuck in a "local basin" of incorrect reasoning.
- Reverse Engineering as Verification: Substitute verification is a powerful "exam skill." By forcing the model to solve for an input variable, the researchers create a different reasoning task that is less likely to suffer from the same systematic bias as the original forward-solving task.
- Efficiency: For simpler problems, the model usually finds the correct answer in 1-2 iterations. For complex problems with irrelevant context, it utilizes more iterations (averaging ~4) to filter out errors.
Limitations and Future Work
While effective, PRP increases inference costs due to its iterative nature, requiring multiple LLM calls per problem. Additionally, the success of the verification step depends on the model's ability to perform reverse calculations. Future work may explore applying this iterative rectification logic to non-mathematical reasoning tasks like code generation or logical puzzles.
Summary updated