When a language model is trained to be safe or to reason at length, it often gets worse at other things. People call this the alignment tax. The fixes proposed so far protect the skills that are easy to measure, such as grade-school math, basic code and common sense. We argue that the most expensive part of the tax sits somewhere nobody is checking: a model's willingness to say "this cannot be answered" on expert questions in physics, electrical engineering or finance. There, the model often lacks the relevant theorem or the problem is missing a needed quantity. Losing that willingness is a direct route to hallucination, meaning confident, fluent, wrong answers. Chain-of-thought reasoning, where the model writes out intermediate steps before answering, makes this harder to catch, because the written reasoning makes wrong answers look more certain.
The proposal has two parts. First, we build a controlled test of whether safety alignment and reasoning training erase expert-domain abstention. Second, we test a specific mechanism. Current "null-space" safety methods freeze the directions in a model's activations used by generic skills. We ask whether also freezing the directions the model uses when it correctly abstains on expert problems preserves abstention without adding any abstention data to the safety training. If the audit finds no such loss, that negative result is itself useful, since it would tell practitioners the generic protections are enough.
Core Claim
We make three claims, one of which is testable in two outcomes. The alignment tax measured on generic benchmarks understates the damage in expert domains, because the damage is mostly epistemic: the model keeps solving answerable theorem problems at roughly the same rate but stops abstaining on unanswerable ones. Current geometric protections cannot see this, because their calibration sets contain no expert-domain abstention behavior. Adding the activations of correct abstentions on expert problems to the protected set should either close this gap or show that abstention is not stored in the directions these methods protect.
Four strands of prior work motivate the claim. The Hallucination Tax of Reinforcement Finetuning shows that reinforcement fine-tuning for math collapses refusal on unanswerable problems. It repairs this by mixing about 10% synthetic unanswerable data into training, but only in synthetic math. Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable shows that blunt refusal training roughly halves reasoning accuracy in a reasoning model, but it measures only harmfulness, not epistemic safety. TheoremQA shows that open models score below 15% on theorem-driven STEM questions and that putting the theorem in the prompt does not help much. This is exactly the regime where models must either operationalize a principle they may not hold or decline. Chain-of-Thought Prompting Obscures Hallucination Cues finds that chain-of-thought lowered hallucination-detection AUROC (area under the ROC curve, a threshold-free measure of how well a detector separates correct from incorrect answers) in 465 of 768 configurations. That result is why we measure abstention behavior directly rather than trusting a detector after the fact.
Research Program
The benchmark. The first artifact is a paired expert-domain abstention set built from TheoremQA's 800 questions. For each question we create up to two unanswerable variants, following the construction style of the SUM dataset in the Hallucination Tax paper. In the first kind, a quantity the theorem needs is deleted. In the second kind, a constraint is added that contradicts the theorem's preconditions. Each variant is checked by two annotators with domain background, and only variants both judge unanswerable are kept. The set is split by theorem, not by question, so the protection method never sees a theorem that appears at test time.
To measure the opposite failure, we add a small set of benign expert questions whose wording resembles dangerous requests, such as circuit fault analysis or chemical kinetics. Alongside it we use ORBench, an existing suite of benign prompts designed to catch over-refusal.
The audit. We train a ladder of checkpoints from two instruction-tuned models, Llama3-8B-Instruct and Qwen2.5-7B-Instruct (the models Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization used), plus one distilled reasoning model of the kind studied in SafeChain. Each starting model receives three safety treatments:
- Direct refusal fine-tuning, which trains the model to refuse harmful prompts outright.
- SafeChain fine-tuning, which trains on 40,000 long, reasoned safe responses.
- NSPO, short for Null-Space constrained Policy Optimization. It builds a matrix of activations from 1,000 generic reasoning prompts and allows only safety updates that leave outputs on those prompts unchanged.
For every checkpoint we measure four things on the paired set: accuracy on answerable items, the abstention rate on unanswerable items, the false-abstention rate on answerable items, and over-refusal on the benign-but-alarming items. Each is measured with and without chain-of-thought.
The mechanism test. The second artifact is a change to NSPO's protected set. NSPO keeps a layer's outputs fixed on a calibration set by constraining each weight update. Let $W$ be a layer's weight matrix, let $\Delta W$ be the proposed update, and let $U$ hold the top singular vectors of the matrix of that layer's inputs on the calibration prompts. The update is replaced by
$$\Delta W \leftarrow \Delta W\,(I - U U^\top),$$
so that $\Delta W x \approx 0$ for any calibration input $x$.
NSPO fills this calibration set with generic prompts. We add two more sources:
- activations from expert-domain answerable problems the model solves correctly;
- activations from the model's own correct abstentions on unanswerable expert variants, captured at the tokens where it writes out the missing premise and declines.
The safety reward, the data and the number of training steps are otherwise identical to NSPO.
A second part of the analysis uses the probing approach of Refusal Falls off a Cliff. That paper found a small set of attention heads that suppress harmful-intent refusal just before the final answer. We check whether these same heads also suppress epistemic abstention, which would explain why safety training and abstention interact at all.
The obvious risk. Every direction added to the protected set shrinks the space left for safety updates. If safety then weakens, the method trades one tax for another. We report the size of the remaining null space and the attack success rate (the fraction of adversarial prompts that get a harmful answer) together.
Baselines
Naive reference. The unmodified instruction-tuned model and its directly refusal-trained counterpart. These show how large the tax is with no protection. Safety Tax reports that direct refusal cut harmfulness to 0.80% while average reasoning accuracy on AIME24, GPQA and MATH500 fell from 63.40% to 32.49%. We expect a comparable collapse to show up as lost abstention on our set.
Strongest existing geometric method. NSPO with its original generic calibration set, run on the same PKU-SafeRLHF subset, with the same GRPO training steps, the same learning-rate search and the same seeds as our variant. NSPO reports an attack success rate as low as 0.18% on Llama3 with general math and code within 1% of the base model. It is the strongest competitor because it already claims to remove the tax. Our variant beats it if it keeps abstention on held-out-theorem unanswerable items within five points of the pre-alignment model while NSPO does not, with attack success rate on HarmBench and JailbreakBench within half a point of NSPO's.
Strongest data-based method. Mixing 10% of our expert unanswerable variants into safety training, following The Hallucination Tax of Reinforcement Finetuning. In synthetic math, that recipe raised refusal from about 0.01 to about 0.7–0.8 at a cost of 0.01–0.05 accuracy on solvable problems. For the unanswerable expert-domain setting: no reported result found on Lacuna. It uses the same abstention items our protected set uses, so both methods see the same information.
Ablation of the key ingredient. Three versions of the protected set separate our two additions: generic only (NSPO), generic plus expert answerable, and generic plus expert answerable plus abstention states. If the third adds nothing over the second, the claim that abstention lives in its own directions is wrong.
Reference points for scale. On TheoremQA, GPT-4 with Program-of-Thought prompting scores 51% and open models score below 15%.
Why not just…?
Why not just mix unanswerable expert data into training, as the Hallucination Tax paper does? This is the real threat, and we run it as a full baseline. Its reported weakness is that the trade-off worsens sharply at higher mixing ratios: a 50% mix clearly hurt answerable accuracy. It also requires every team doing safety alignment to curate abstention data for every domain it cares about. The protection method needs only unlabeled activations of abstentions the model already produces. If 10% mixing matches our variant on both abstention and answerable accuracy, we will say so, and the audit becomes the main contribution. The deciding case is a domain held out of the mix: if protected directions carry over to held-out theorems and mixed data does not, the geometric route earns its place.
Why not just add expert problems to NSPO's calibration set? This is our second ablation arm, and it may be enough. Our hypothesis is that answering and declining use partly different directions, so protecting correct solutions does not protect correct abstentions. The comparison between the second and third arms settles this directly.
Why not just detect hallucinations at inference time and abstain when the detector fires? HARP projects hidden states onto the roughly 5% of the output-layer geometry left after removing the dominant semantic directions. It reaches 92.8% AUROC on TriviaQA with Qwen-2.5-7B. However, it was never tested on theorem-driven problems, and chain-of-thought degrades such detectors in most settings. We train HARP on each checkpoint, apply a threshold abstention rule, and report its AUROC on our set with and without chain-of-thought. If it recovers abstention after alignment, it is a cheaper alternative, and we will report it as such.
Why not just put the theorem in the prompt? TheoremQA found that supplying the theorem gave no significant accuracy gain. For our unanswerable variants, the missing piece is a quantity in the problem, which no retrieved theorem can supply. We include a theorem-in-prompt condition only to confirm that it does not change abstention.
Proposed Experiments
Decisive check 1: is there a loss to prevent? This comes first. If NSPO-aligned checkpoints keep expert-domain abstention within five points of the pre-alignment model, the premise fails and we stop the method work. If abstention drops while answerable TheoremQA accuracy and MATH500 hold steady, the loss is real and was invisible to existing evaluations.
Decisive check 2: does protection work? At matched attack success rate, we compare abstention on held-out-theorem items for the three protected-set arms and the 10% mixing baseline. We also report false abstention and over-refusal on ORBench, so that a method which simply declines more often cannot win.
Decisive check 3: do the two kinds of refusal share machinery? We ablate the refusal-suppressing heads found by the Refusal Cliff probing. If ablating them also restores epistemic abstention in the reasoning checkpoints, harmful-intent refusal and "I don't know" share a mechanism.
Generalization. SUM, UMWP (unanswerable math word problems) and SelfAware (a factual abstention set) test whether expert-domain protection carries back to generic abstention. All results are reported with and without chain-of-thought, because the obscuring effect found in the CoT-and-hallucination study predicts that the gap between the arms will be largest when the model reasons aloud.
References
- The Hallucination Tax of Reinforcement Finetuning
- Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
- TheoremQA: A Theorem-driven Question Answering Dataset
- Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
- Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
- SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities
- Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
- HARP: Hallucination Detection via Reasoning Subspace Projection