Teaching Models When Not to Trust a Theorem: Transferring Premise Rejection from Math to Expert Domains

Reinforcement learning on math problems with checkable answers makes language models better reasoners in other fields too, including code and graduate-level science. Every one of those transfer results was measured on clean prompts. Real expert use looks different. A model answering a physics or finance question is usually handed retrieved text, and that text often contains a theorem that is nearly right: a wrong constant, a missing condition, a neighbouring law. There is evidence that models which "think" longer are more, not less, willing to build an argument on such a bad premise. This note proposes to test whether math reinforcement learning causes that weakness, and whether a cheap change to the math training data can remove it in domains the model was never trained on.

The specific idea is to mix two kinds of injected context into math training problems, where correctness can be checked automatically. Some problems come with a plausible but wrong formula. Others come with a definition the problem cannot be solved without. The model is rewarded only for the correct final answer. We then ask whether the resulting skill, using context when it is needed and rejecting it when it is wrong, carries over to university-level physics, electrical engineering and finance theorems. The evaluation is generative throughout. Answers are free-form numbers or executed programs, and the model must state which supplied premise it relied on, so it cannot score by eliminating options.

Core Claim

We make two linked claims, each of which can be falsified. The first is diagnostic. Standard math reinforcement learning with verifiable rewards improves expert-domain accuracy when no context is supplied, but it increases the harm rate under misleading theorem context. We define the harm rate as the fraction of questions a model solves with no context that it gets wrong once a corrupted theorem is supplied. The mechanism we suspect is simple. Math training sets almost never contain wrong premises, so RL rewards using everything given.

The second claim is constructive. If a fraction of math training problems carry either a corrupted formula or an indispensable novel definition, the model learns a premise-checking habit. That habit should lower the harm rate on held-out expert domains without reducing the benefit of correct context.

The closest prior works each cover part of this. AbstRaL trains models to rewrite word problems into symbolic form. This keeps irrelevant numbers out of the calculation on GSM-Plus (Distract), which adds topic-related but useless sentences to math problems. A wrong formula, however, would be abstracted faithfully, and AbstRaL is evaluated only on math. Thinking in a Crowd documents what it calls the rationalization trap, a model reasoning its way into accepting a bad premise, but proposes no fix and studies school science. Our proposal differs from both. The noise it targets is misleading rather than irrelevant. The fix is trained where answers can be checked and tested where they cannot be checked during training.

Why This Gap Is Real

Three findings motivate the work. CPL and Reasoning Vectors show that math-only RL transfers outward. CPL trains high-level plans with tree search and reports +8.6% on GPQA. Reasoning Vectors adds the weight difference between RL-trained and supervised checkpoints to another model and reports +12.3% on BBH for Qwen2.5-1.5B. Both measure on clean prompts. Thinking in a Crowd reports that Qwen3-8B falls from 96.22% clean accuracy to 77.03% with two misleading snippets when thinking is on. With thinking off it reaches 74.87%, so thinking recovers only about two points of a nineteen-point loss even though it adds roughly 10–13% on clean questions. Reasoning traces also grow much longer under misleading context.

TheoremQA contains 800 questions built on 350 theorems across math, physics, EE&CS and finance. GPT-4 reaches 51.0% with Program-of-Thought prompting, in which the model writes code that is executed to get the answer. Open-source models score below 15%. Pasting the correct theorem into the prompt gives no significant gain.

Two findings constrain the design. BRIGHT shows that retrieval for theorem-style queries is poor: the best dense model averages 22.5 nDCG@10. A realistic pipeline therefore hands the reader near-miss theorems. We use exactly this to build realistic noise. TheoremQA's null result for theorem augmentation warns that "give it the right theorem" is not a solution. Our metric therefore separates the gain from correct context from the harm from wrong context, rather than reporting a single averaged number.

Research Program

Evaluation suite: Noisy-TheoremQA. For each of TheoremQA's 800 questions we build five context conditions:

  • no context;
  • the gold theorem statement;
  • a corrupted gold theorem, with a changed constant, a dropped condition or a swapped variable, generated by an LLM and checked by hand on a 150-question subset;
  • an irrelevant theorem from a different TheoremQA domain;
  • the BM25 top-5 passages from BRIGHT's TheoremQA theorem split (xlang-ai/BRIGHT), which supplies naturally occurring near misses.

Answers are produced by Program-of-Thought and checked numerically. The model must also output which supplied statement it used, or that it used none. This attribution can be checked because we know which statement was injected.

The headline metrics are three:

  • harm rate, defined above;
  • gold gain, the accuracy with the gold theorem minus accuracy with no context;
  • attribution accuracy, the share of questions where the model correctly says which statement it used.

Reasoning length is reported as a secondary metric. The SciAux misleading-context split (billhdzhao/SciAux) serves as a second, out-of-distribution test.

Diagnostic study. We use matched Qwen2.5 checkpoint pairs, a supervised one and an RL-trained one, in the style of Reasoning Vectors. Adding and subtracting the reasoning vector traces how harm rate moves as RL-acquired reasoning is dialled up and down. Qwen3-8B with thinking on and off gives a second axis.

Training intervention. Premise-contrastive RL uses GRPO, a reinforcement learning method that scores each answer against a group of sampled answers to the same problem. It runs on GSM8K and MATH. Each training problem is assigned to one of three cases:

  • Clean: about 50% of problems have no injected context.
  • Corrupted premise: about 25% carry a corrupted but relevant formula, such as a wrong area formula or a misstated identity. The correct answer requires ignoring it.
  • Necessary premise: about 25% depend on a novel defined operator or quantity that appears only in the context, so ignoring the context guarantees failure.

The reward is answer correctness, plus a small bonus for correct premise attribution. The necessary-premise problems stop the model from learning the trivial policy of ignoring all context, which would show up as a drop in gold gain.

Baselines

The naive reference is the supervised Qwen2.5 checkpoint, prompted with Program-of-Thought, under all five conditions. It sets the floor for every metric. There is no reported result on Lacuna for TheoremQA under injected noise for these models, so this floor must be measured.

The strongest existing transfer method is standard math GRPO with the same problems, steps and sampling budget, but no injected context. This is the RL checkpoint behind Reasoning Vectors, whose reported out-of-domain gain is +12.3% on BBH for Qwen2.5-1.5B. It is the true competitor because it isolates whether our data change matters beyond RL itself. Our method beats it only if harm rate drops while gold gain and no-context accuracy stay within one standard error of it across three seeds.

The strongest inference-time noise defence has two members, both run on the same base checkpoint with matched decoding budgets:

  • Distraction-aware answer selection from Improving Zero-shot Reader. It combines an "unanswerable" escape prompt with a query-likelihood penalty and reports a 64% average Exact Match gain over naive readers in open-domain QA.
  • Ext2Gen from Aligning Extraction and Generation. It trains the model by preference optimization to extract evidence sentences before answering, and reports a QA score of 0.557 against 0.473 for Llama3.1-8B-Instruct under noisy retrieval. We reproduce Ext2Gen's training on its own data, applied to our base model, so that it is not handicapped.

A reasoning-model reference is Qwen3-8B with thinking on, anchored to the SciAux figure of 77.03% under misleading snippets.

Ablations isolate the key ingredient:

  • irrelevant-only distractors in training, which approximates what AbstRaL-style robustness targets;
  • corrupted premises without the necessary-premise cases;
  • removal of the attribution bonus.

If irrelevant-only distractor training matches the full method on corrupted-theorem harm rate, our claim that the corrupted premises carry the transfer is wrong.

Why not just…?

Why not just tell the model to ignore unreliable context, as distraction-aware selection does? This is a real threat. On SciAux-style misleading snippets, a well-worded escape prompt may close much of the gap. Our expectation is that corrupted theorems are maximally on-topic, so a query-likelihood penalty will score them as relevant and the escape prompt will rarely fire. The direct comparison on the corrupted-theorem condition settles this. If prompting matches trained premise rejection, we would report that and the training contribution would shrink to the diagnostic finding alone.

Why not just use Ext2Gen-style extract-then-generate? Extraction picks sentences that look relevant. A theorem with a flipped sign looks exactly as relevant as the true one, so we expect Ext2Gen to help on the irrelevant and BM25 conditions but not on the corrupted one. The per-condition breakdown tests this, and a combination of Ext2Gen and our training is reported too.

Why not just use symbolic abstraction like AbstRaL? AbstRaL removes distractor quantities by mapping them to variables. A misstated formula, however, is carried into the symbolic program exactly as written. AbstRaL also relies on teacher-generated decomposition traces and an external solver. The irrelevant-only ablation stands in for this family of methods on our metric.

Why not just train on noisy expert-domain data directly? Expert domains lack automatic answer checkers, which is the bottleneck that motivates transferring from math. As an upper bound, we train on a held-out half of TheoremQA with injected noise and test on the other half. This shows how much of the in-domain ceiling math-only training recovers.

Why not just use a reasoning model with longer thinking? SciAux already shows thinking-on Qwen3-8B dropping to 77.03% under misleading context, only marginally above thinking-off (74.87%), with traces growing much longer. We include it as a reference rather than a solution.

Decisive Checks

Three results would decide the project:

  • Diagnostic claim. Harm rate on corrupted theorems should rise monotonically as the reasoning vector is added, while no-context accuracy also rises. If harm rate is flat or falls, the diagnostic claim is refuted.
  • Constructive claim. Premise-contrastive training should cut corrupted-theorem harm rate relative to standard GRPO, with gold gain not lower. The reduction should appear in the physics and finance subsets, where no training problem resembles the content. Gains confined to the math subset would mean the skill did not transfer.
  • Noise-type specificity. The irrelevant-only ablation should fail to reduce corrupted-theorem harm. Without this result, "premise rejection" collapses into ordinary distractor robustness, which is already known.

Attribution accuracy gives a direct, generative check that lower harm comes from rejecting the bad premise and not from ignoring context altogether. A model that ignores all context would show a falling gold gain and attribution answers of "none" on the gold condition.