Language models answering math word problems often change their final answer when the prompt is reworded in a way that should not matter, such as a paraphrased instruction, an added "thank you," reordered sentences, or renamed variables. They also make plain arithmetic mistakes, like getting a four-digit multiplication wrong. Separately, models can now read hundreds or thousands of worked examples in a single prompt ("many-shot in-context learning," meaning the model learns a task from examples in its input rather than from weight updates). This proposal asks one question that joins the three: when a many-shot math prompt flips its answer under a harmless rewording, is the flip caused by the model setting up the problem differently, or by the model doing the same setup and then miscalculating? The design runs the same model on the same examples in three conditions: free-form reasoning, reasoning in which every arithmetic step is checked and corrected by a calculator during generation, and reasoning written as a Python program that an interpreter executes. Comparing answer flips across the three conditions, and across shot counts from 0 to about 1,000, splits brittleness into a setup part and an arithmetic part.
This matters because two known failures of many-shot math prompting have no explanation yet. Many-Shot In-Context Learning reports that accuracy on the MATH benchmark starts to degrade after roughly 125 shots. Brittlebench reports that adding a few examples makes models more sensitive to formatting than using none. If both effects come mostly from arithmetic, the fix is cheap: hand the arithmetic to a tool and keep scaling the examples. If they come from problem setup, tools will not help, and the issue lies in how long contexts shape the model's reading of the problem. Whichever way it comes out, the result tells practitioners how many examples to use and whether tool use makes those examples safe.
Core Claim
The proposal makes a claim that can be tested and could turn out false. In many-shot chain-of-thought prompting on arithmetic-heavy problems (chain-of-thought means the model writes out its reasoning before the answer), we expect the share of prompt-induced answer flips caused by computation to grow with shot count, and the MATH overloading effect to shrink sharply once arithmetic is corrected during decoding. The setup part of brittleness, by contrast, should fall as examples accumulate. That would match the finding of Revisiting In-Context Learning with Long Context Language Models that large random example sets cover most of a task's input distribution and make selection and label noise matter less. The new contribution is not a new prompting method. It is a controlled decomposition of prompt sensitivity into setup and computation, made possible by calculator intervention at decoding time, and measured across the full range of shot counts.
To make this precise, fix a problem and a set of $K$ meaning-preserving rewrites of its prompt. Call the problem ambiguous under a condition if the final answers across the $K$ rewrites are not all identical. We then report the dataset-level ambiguity rate $A$, following Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity. With $A_{\text{free}}$ for free chain-of-thought and $A_{\text{calc}}$ for calculator-corrected chain-of-thought, both run on the same examples, the computational share is
$$ S_{\text{comp}}(n) = \frac{A_{\text{free}}(n) - A_{\text{calc}}(n)}{A_{\text{free}}(n)}, $$
where $n$ is the number of in-context examples. Ambiguity is not strictly additive, since correcting the arithmetic can also change which setup the model continues with. For that reason we also report the overlap between the sets of ambiguous problems under each condition, not just the difference in rates.
Research Program
The key tool is a calculator that intervenes during generation, adapted from the calculator annotations in Training Verifiers to Solve Math Word Problems. Every demonstration writes each arithmetic step in a tagged form. At test time, whenever the model closes an expression tag, decoding pauses, the expression is evaluated exactly, and the correct value is inserted before generation continues. Because the model then conditions on the correct intermediate value, errors do not cascade downstream. Only arithmetic is removed from the model's control; the prose setup is untouched.
The program-aided condition follows PAL: Program-aided Language Models, which has the model write Python with descriptive variable names and runs it afterwards. Comparing the calculator condition with the program condition separates two things: whether the setup is written in prose or in code, and whether arithmetic is computed or guessed.
All three conditions use demonstrations built from the same problems. The rationales are generated by the model and kept only if correct, in the "reinforced" style of Many-Shot In-Context Learning. Each kept solution is stored in three renderings: free prose, prose with tagged arithmetic, and a program. Any difference between conditions therefore reflects the format, not which problems were chosen as examples. Examples are sampled at random and held fixed across test queries. This follows the finding in Revisiting In-Context Learning with Long Context Language Models that random selection matches clever selection beyond 64 shots, and it lets the attention cache for the example block be reused across queries. Shot counts run over 0, 1, 4, 16, 64, 125, 256, 512 and 1,024. We record the fraction of the context window used, because the same paper reports performance peaking near 25% fill, and that effect must not be mistaken for overloading.
The prompt rewrites cover three families.
The first family uses Brittlebench's instruction-level categories: word-level noise, polite or persona additions, formatting wrappers, and paraphrased instructions. These are applied to the instruction and the test question only, with the examples fixed.
The second family changes the story of the problem itself, in the style of SVAMP: reordered sentences and irrelevant details. Rewrites that change the answer are filtered out by an equivalence check.
The third family permutes the order of the examples. Many-Shot In-Context Learning notes that order sensitivity persists at scale. Order rewrites are analysed separately because they break cache reuse.
A small mechanistic check links the behavioural split to the internal "abstract then compute" pipeline found by Can LLMs Reason Abstractly Over Math Word Problems Without CoT?. That paper finds problem formulation in middle layers of Llama-3 8B and arithmetic in later layers, with arithmetic as the main failure point. For problems that are ambiguous under free chain-of-thought but not under the calculator condition, we patch activations from a stable rewrite into a flipping one, first at the formulation layers and then at the computation layers. The prediction is that patching the late layers recovers most flips at high shot counts. This check is confirmatory; the main argument does not depend on it.
Prior Work and Positioning
The closest existing work each pairs two of the three directions, and none varies shot count while measuring brittleness on math.
PAL shows that moving arithmetic into code keeps GSM-HARD accuracy stable. It does this with 3–8 examples and never rewords the prompt. Many-Shot In-Context Learning finds the overloading effect on MATH with prose rationales only and no rewrites. Brittlebench splits performance variance into a difficulty part and a prompt-sensitivity part, and includes MathQA, but stops at about 5 shots. It also reports that chain-of-thought reduces the perturbation drop by only about 17%, which motivates asking which part of chain-of-thought fails. ProSA ties sensitivity to low decoding confidence and finds MATH especially brittle, but covers only 0 and 1 shots. Controlling Equational Reasoning with Prompt Interventions finds predictable error patterns under variable renaming and step removal, on synthetic LaTeX derivations without word problems or tools. We borrow its variable-renaming rewrite.
ASyMOB appears to cut against the proposal. It reports that tool use barely helps the strongest reasoning models, while giving GPT-4o a boost of up to 37.7%. We treat this as a prediction to test rather than an objection. For such models, $S_{\text{comp}}$ should be near zero at every shot count, and any remaining brittleness should be setup brittleness. That would also resolve the apparent tension between ASyMOB's reported robustness and Brittlebench's observed drops in frontier models.
Baselines
The naive reference point is zero-shot chain-of-thought with the standard instruction, together with its ambiguity rate under the same rewrites. Every other curve is read relative to it. Brittlebench and ProSA suggest that zero-shot prompting may be less brittle than few-shot but less accurate, and this reference makes that trade-off visible.
The strongest existing method for arithmetic robustness is few-shot program-aided prompting as in PAL. It reports 72.0% on GSM8K with Codex, against 56.9% for PaLM-540B with chain-of-thought. On GSM-HARD, chain-of-thought falls from 65.6% to 20.1% while PAL stays at 61.5%. We rerun PAL at 8 shots on our models with the same filtered example pool, so that it competes with its many-shot versions on equal terms. Many-shot program-aided prompting beats it only if it lowers ambiguity on GSM-HARD and SVAMP-style rewrites by a margin larger than the spread across 3 seeds of example sampling, without losing accuracy.
The strongest existing many-shot method is reinforced many-shot chain-of-thought from Many-Shot In-Context Learning. The file reports 83% on Big-Bench Hard against 72.1% for 3-shot human-written chain-of-thought, and degradation on MATH past about 125 shots. It gives no GSM8K accuracy; no reported result was found on Lacuna for many-shot chain-of-thought on GSM8K. We reproduce this method as our free chain-of-thought condition on the same examples.
The ablation that isolates the key ingredient is the calculator-corrected chain-of-thought condition. It uses identical examples and identical prose formulation, and differs only in exact arithmetic. If overloading and rising brittleness survive this condition, the computational explanation is refuted.
For the confidence use case, SENSEL abstains when answers disagree across paraphrases. It reports an average gain of +4.1 area-under-curve points (F1 against coverage) for GPT-J-6B on classification tasks, and no math result. We run it with the same number of forward passes as every other ensemble we compare against.
Why not just…?
Why not just use few-shot program-aided prompting and skip many shots? This is the most serious threat. PAL already removes most arithmetic error (PAL). If our decomposition shows that brittleness is mostly computational, a few PAL examples may be all anyone needs. The experiment settles the question directly: if the few-shot and many-shot PAL curves for ambiguity and accuracy lie on top of each other, we report that many examples add nothing once arithmetic is handed off, and treat it as a primary finding. Two reasons suggest the answer is not foregone. PAL depends on the model writing correct code. And Brittlebench finds that few-shot framing increases formatting brittleness, so a program prompt with 8 examples may be sensitive to setup rewrites in a way that many examples smooth out. The contribution survives either result, because it is the decomposition, not a claim that many-shot PAL wins.
Why not just sample several answers and take a majority vote? Voting reduces randomness within a single prompt. It does not address flips across rewordings, and Rethinking Hallucinations shows that consistency-style signals track randomness rather than correctness. We include a compute-matched arm: majority voting over free chain-of-thought with as many samples as there are rewrites. If voting alone drives ambiguity as low as the calculator condition does, the computational explanation is weakened. That would suggest the flips are sampling noise rather than systematic arithmetic failure.
Why not just retrieve a handful of similar examples for each question? Revisiting In-Context Learning with Long Context Language Models finds that query-specific selection beats random selection in fewer than 15% of cases past 64 shots. It also notes that per-query prompts cannot reuse the cached example block. We still run 8 retrieved examples, chosen by embedding similarity, as an arm, because retrieval could reduce setup brittleness on story-level rewrites. However, the retriever itself sees the reworded question, so it may add its own sensitivity.
Why not just fine-tune on math data? Fine-tuning changes the object under study. The question is how the demonstration context shapes brittleness, and Many-Shot In-Context Learning reports rough parity between many-shot prompting and fine-tuning on the same data in other tasks. A LoRA model trained on the same filtered pool is a useful reference point for the budget-matched comparison, and we include it at one scale. It does not replace the shot-count sweep.
Proposed Experiments
We use GSM8K, GSM-HARD, SVAMP and MATH as the main benchmarks. A 2,000-problem subset of ASyMOB, generated with the ASyMOB codebase, serves as the symbolic stress test. Each test problem gets 12 filtered rewrites across the three families. The models are an open-weight long-context family at two scales (8B and roughly 70B), one commercial long-context model, and one reasoning model to test the ASyMOB prediction. Metrics are accuracy, ambiguity rate, computational share, overlap between ambiguous sets across conditions, context fill, and cost in tokens per query with and without cache reuse.
Three results would be decisive.
First, the overloading test. Plot MATH accuracy against shot count under the free and calculator-corrected conditions. If the drop past about 125 shots mostly disappears with the calculator, overloading is an arithmetic effect. If it persists at the same fill fraction in both, it is a setup or attention effect.
Second, the brittleness trend. If $S_{\text{comp}}(n)$ rises with $n$ while calculator-condition ambiguity falls, many examples stabilize setup while degrading computation. That would reconcile Brittlebench's few-shot result with the stabilization seen in Revisiting In-Context Learning with Long Context Language Models.
Third, scale dependence. For the reasoning model, $S_{\text{comp}}$ should stay near zero, which would locate its residual brittleness in setup.
A null result on all three, meaning flat computational share and overloading that survives calculator correction, would still be informative. It would show that tool use cannot buy robustness in long math prompts and would point future work toward how long contexts shape problem formulation.
References
- Many-Shot In-Context Learning
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- Revisiting In-Context Learning with Long Context Language Models
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- Training Verifiers to Solve Math Word Problems
- PAL: Program-aided Language Models
- Are NLP Models really able to Solve Simple Math Word Problems?
- Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Controlling Equational Reasoning in Large Language Models with Prompt Interventions
- ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark
- On the Relation between Sensitivity and Accuracy in In-Context Learning