Language models that solve long math word problems can still stumble when a key quantity is stated through an idiom. "They went Dutch on the bill" means the cost was split evenly. "A baker's dozen" means thirteen. "The company broke even" means profit was zero. In each case, the meaning of the phrase cannot be read off its individual words. This proposal asks a narrow, testable question. When a reasoning problem states a load-bearing constraint through a multi-word expression (a phrase whose meaning is more than the sum of its words, such as an idiom, a fixed collocation, or a multi-word technical term), how much does accuracy drop? And is that drop caused by the model misreading the phrase itself, or by a general sensitivity to rewording? We build matched pairs of problems that differ only in whether one constraint is stated literally or through such an expression. We then test whether rewriting just that span into plain language repairs the failure better than rewriting the whole problem.
The opportunity comes from two findings in separate literatures that have not been joined. Idiom research shows that models understand idioms partly by memorization, and that step-by-step prompting sometimes hurts idiom comprehension. Reasoning research shows that "thinking" modes can build long, confident justifications for a misread premise. If both hold, a reasoning model that fails to recognize an idiom should do more than get it wrong. It should reason carefully from the literal reading. That makes it a concrete, measurable failure mode rather than generic "fragility."
Core Claim
We claim three things. First, stating a load-bearing constraint through a non-compositional multi-word expression causes a larger accuracy drop than a literal paraphrase of equal length. Second, the drop is concentrated in expressions that are both low in compositionality and poorly memorized, the two properties that Memorization or Reasoning? Exploring the Idiom Understanding of LLMs found drive idiom comprehension. Third, a span-local intervention repairs most of the drop while changing nothing else in the problem. The intervention detects the expression, replaces only that span with a plain paraphrase, and checks that the meaning was preserved.
The part that would be new is the separation of two kinds of expression. Load-bearing expressions encode an operation or quantity the solution depends on. Decorative expressions are flavor text, the "redundant information" failure mode that Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? identified but did not break down linguistically. We also claim that, for load-bearing expressions, thinking modes increase rather than reduce the literal-reading error.
Relation to Prior Work
Two papers motivate the gap. Memorization or Reasoning? builds MIDAS, 64,660 idioms in six languages, and tests comprehension with five-way multiple-choice questions. It finds that models lean on memorization and fall back on composing word meanings, and that Chain-of-Thought (step-by-step prompting) sometimes degrades accuracy. It never places idioms inside a downstream reasoning problem and proposes no intervention. We reuse its memorization and compositionality measures as the explanatory axes of our study.
Analyzing the Role of Semantic Representations in the Era of Large Language Models injects Abstract Meaning Representation graphs (structured semantic parses) into prompts. Its paraphrase-detection sets built around slang and multi-word expressions drop by 6.63 to 8.78 points. This constrains our design: it is evidence against routing phrases through a symbolic intermediate, so our intervention stays in plain text.
Thinking in a Crowd supplies the mechanism we test for. On SciAux, two misleading snippets drop Qwen3-8B to 77.03% with thinking on, against 74.87% with thinking off and 96.22% without the snippets. Response length peaks during rationalization. Its misleading content comes from external snippets. Ours comes from the model's own misreading of a phrase in the problem itself.
On the simplification side, Multi-Word Lexical Simplification defines the task of replacing 1–3 word fragments. Its system, Plainifier, generates candidates with a BERT variant that has a learned "empty" token, and ranks them by contextual probability, embedding similarity and word familiarity. It reports that ranking, not generation, is the bottleneck, and it evaluates only against human substitution lists, never on downstream tasks. We borrow its framing of span-local replacement but judge replacements by whether reasoning survives.
The tokenizer study BPE beyond Word Boundary finds that adding frequent n-grams to the vocabulary hurts translation (22.0 to 20.7 BLEU on Hindi-English), while filtering by pointwise mutual information (a measure of how much more often words co-occur than chance) recovers it (22.1). We reuse that statistic to filter candidate spans in our detector.
Two further papers supply tools and constraints. ProcessBench shows that 51.8% of correct-answer solutions on its hardest math tier contain intermediate errors. This warns us that final accuracy alone can hide literal misreadings, so we also localize the first wrong step. When Splitting Makes Stronger shows that decomposing a problem into sub-tasks helps when the sub-tasks are disjoint. That condition an idiom split across sub-tasks would violate, so we do not use decomposition as the intervention.
Research Program
Benchmark construction. We start from the 298 math pairs of ConsisEval, derived from GSM8K, and from additional GSM8K problems. For each problem, annotators pick one constraint the solution depends on, for example "split the cost equally," "doubled," or "twelve plus one." They rewrite it into a multi-word expression with the same meaning, drawn where possible from MIDAS entries whose figurative definitions match.
Each item then exists in four matched versions:
- the original literal problem;
- a literal rewording of the same span, matched for length, which controls for rewording in general;
- a load-bearing expression version, where the constraint is stated through the idiom;
- a decorative version, where an unrelated idiom is inserted as flavor text and the literal constraint is kept.
Two annotators verify that each version has the same answer, and disputed items are discarded, following the consensus filtering ProcessBench applies.
Each expression is tagged with three properties. Compositionality uses MIDAS-style 1–5 inferability ratings. Memorization uses the MIDAS idiom-completion probe run on each evaluated model. The third tag marks whether the expression is a figurative idiom or a multi-word technical term such as "least common multiple." Technical terms are the case where simplification might harm reasoning by blurring a precise concept. A smaller Turkish and Korean slice, built from MIDAS idioms, tests the low-memorization regime MIDAS found in those languages.
Failure analysis. For every wrong answer, we check whether the first erroneous step is the step where the expression is interpreted. We also check whether the trace reads the phrase literally. A strong critic model first labels the first error, in the style of ProcessBench, and then humans audit a stratified sample. This tests the literal-bridge hypothesis directly, instead of inferring it from accuracy.
Intervention. The span-local simplifier runs in three steps:
- Detect. A language model proposes candidate multi-word spans, which are filtered by pointwise mutual information on a reference corpus.
- Rewrite. The model rewrites only each detected span, conditioned on the whole problem.
- Check. The model answers a small set of constraint questions about the problem both before and after rewriting, covering quantities, operations and who does what. A rewrite is rejected if any answer changes.
The check is our response to the ranking bottleneck that Multi-Word Lexical Simplification reported: instead of scoring fluency or familiarity, we score whether the constraints survive.
Baselines
Unmodified problem. The first reference point is the model answering the load-bearing expression version as written, without and with Chain-of-Thought, and for Qwen3-8B with thinking off and on. The gap between this condition and the literal original is the effect we are trying to repair. The gap between the literal original and the length-matched literal rewording is the noise floor from rewording alone. On clean inputs, the numbers to anchor against are ConsisEval's reported consistency scores: 92.2% for GPT-4 Turbo and 91.9% for Llama3-70B-Instruct, the probability of solving the easy item given the hard one is solved. We will report how that score changes when only the easy item is rewritten with an expression.
Whole-problem plain-language paraphrase. The strongest competitor is simply asking the same model to "restate this problem in plain, literal English" and then solving the restatement. It needs no detector and could repair idioms as a side effect. No reported result was found on Lacuna for whole-problem paraphrasing as a pre-reasoning step on math word problems. It will run on the same model, with the same number of extra calls and tokens as our pipeline, and with prompts tuned on the same held-out development split. We beat it if two conditions hold together. Span-local simplification must match or exceed its accuracy on load-bearing items. It must also change strictly fewer non-target constraints, as measured by the constraint-question check and a human audit of 200 rewrites. If paraphrase matches us on both, our intervention adds nothing, and we will report that.
Definition glossing. The third baseline keeps the expression and appends its dictionary definition from MIDAS, using the same detector. This isolates whether replacing the phrase matters beyond simply supplying its meaning. It matters because Memorization or Reasoning? reports that added context consistently helps idiom comprehension.
Plainifier. For the simplification component alone, we run Plainifier as the non-LLM span rewriter. Multi-Word Lexical Simplification reports that it beats dictionary baselines on MWLS1, but the file gives no exact numbers for it.
Ablations. Four ablations isolate the key ingredients:
- rewriting random non-expression spans of equal length, which tests whether any local edit helps;
- skipping the constraint check, which tests whether verification matters;
- rewriting technical terms as well as idioms, which tests harm from over-simplification;
- an oracle that substitutes the original literal text, which gives the upper bound.
Why not just…?
Why not just rely on thinking mode or Chain-of-Thought? The existing evidence points the other way. MIDAS reports that step-by-step prompting sometimes lowers idiom comprehension. SciAux shows thinking mode doing worse than no thinking once a premise is misleading. Our thinking-on versus thinking-off comparison on load-bearing items, paired with literal-reading annotations of the traces, settles this directly. If thinking mode closes the gap, the literal-bridge hypothesis is false.
Why not just paraphrase the whole problem? This is a real threat. For short problems, a capable model may paraphrase faithfully, and our advantage would disappear. Our bet is that whole-problem rewriting sometimes quietly alters numbers, units or relations. That would be the rewrite-level version of the hidden process errors ProcessBench documents. The constraint-drift measurement above is designed to show whether this happens. If drift is negligible and accuracy is equal, the contribution narrows to the diagnostic benchmark and the compositionality-by-memorization analysis. That analysis stands on its own.
Why not just add a gloss instead of replacing the phrase? A gloss is cheaper and keeps the original wording. However, it leaves the misleading literal words in the prompt, where a reasoning model may still latch onto them. The gloss baseline answers this empirically. We will also report where on the compositionality scale glossing and replacement diverge.
Why not just fix the tokenizer so idioms are single units? That is impossible for closed models. Frequency-based merging also measurably hurt translation in BPE beyond Word Boundary. Tokenization also cannot explain failures on idioms that are memorized as strings but misapplied, which our memorization probe separates out.
Proposed Experiments and Decisive Checks
The evaluation covers Qwen3-8B with thinking on and off, Llama3-70B-Instruct, and QwQ-32B. For each model, the benchmark measures three things: accuracy on the four problem versions, consistency score on the rewritten ConsisEval pairs, and first-error localization.
Our core claim fails if any of three things happens:
- load-bearing items lose no more accuracy than the length-matched literal rewordings;
- the loss does not grow as compositionality and memorization fall;
- first errors do not concentrate at the step where the expression is interpreted.
The intervention claim fails if span-local simplification does not beat whole-problem paraphrase on the combination of accuracy and constraint preservation, or if the random-span ablation recovers the same gains.
A secondary check applies the same pipeline to SciAux questions in which a scientific term or figurative device is a multi-word expression. This tests whether local simplification also reduces the rationalization-driven drop reported there. Finally, the Turkish and Korean slice tests whether the benefit is largest where MIDAS found memorization weakest. If it is, local simplification is a practical fix for the language gap in reasoning.
References
- Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- Analyzing the Role of Semantic Representations in the Era of Large Language Models
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Multi-Word Lexical Simplification
- BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation
- ProcessBench: Identifying Process Errors in Mathematical Reasoning
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs