Screening, Not Reflecting: Single-Pass Repair of LLM Over-Refusal

When a language model's safety training misfires, it refuses obviously benign requests — "kill a Python process", or, worse for science, "compute the reaction barrier for a toxic drug metabolite". Two fixes exist today and each fails in an opposite way. Think-Before-Refusal (Wang et al.) makes the model write out a reasoning block before answering, which repairs false refusals but pays a multi-stage verification tax in tokens and latency on every single query. ProSafePrune (Sun et al.) instead surgically removes a fixed "danger subspace" from the model's weights, costing nothing at runtime — but it is blind to context, tuned once on static prompt sets, and cannot tell a toxicology audit from a weapons request.

This proposal imports a strategy from computational chemistry, where an analogous problem was solved decades ago. Simulating thousands of explicit water molecules around a molecule is prohibitively slow, so implicit solvation replaces them with a smooth dielectric field that screens electrostatic interactions; solving the mutual polarization between molecule and field iteratively is still costly, so conductor-like models (COSMO, in Steinmann et al.) solve the easy limit once and apply a closed-form correction — turning an iterative problem into a one-shot operator. We propose doing exactly this for safety: pay for iterated safety reflection once, offline, and distill it into a small, continuous, input-dependent screening operator that runs inside the model in a single forward pass.

Core Claim

Over-refusal is caused by "over-harmful encoding": benign-but-scary prompts get projected into the same low-rank subspace as genuinely malicious ones in middle layers (Sun et al.). Prior work removes that subspace with a hard, fixed projection. We claim the right operation is instead a smooth, per-prompt screening strength — the latent analogue of a dielectric screening function — and that this strength can be learned by distilling an iterated reflection process into a one-pass predictor. The result should match reflection-based repair in compliance while generating almost no extra tokens, and beat static pruning exactly where context matters: the borderline, profession-adjacent queries where current guardrails fail most.

Mechanism: Reaction-Field Screening

Let $\Omega$ be the rank-$r$ projector onto the danger subspace, estimated once by truncated singular-value decomposition of middle-layer activations, following Sun et al.. ProSafePrune permanently rewrites weights as $W' = (I - \lambda\Omega)W$ with one global constant $\lambda$ for every input. We instead intervene on activations at inference with an input-dependent strength:

$$h' = h - \lambda(h)\,\Omega\,h,\qquad \lambda(h)\in[0,1]$$

where $h$ is a middle-layer activation and $\lambda(h)$ is predicted by a lightweight head reading that same activation. The value $\lambda = 1$ is the "conductor limit" — complete subspace removal, ProSafePrune's behavior — and $\lambda = 0$ leaves the representation untouched; everything between damps the danger component smoothly. This mirrors how COSMO replaces an iterated dielectric solve with the conductor limit plus a scaling function of $(\varepsilon - 1)/(\varepsilon + k)$ (Steinmann et al.): the hard solution is the easy special case, and the learned correction adapts it to the actual input. The design also takes seriously a caution from the physics literature: sharp boundaries at the solute–solvent interface cause numerical instability and unphysical cavities, which smoothed interfaces fix (Chai and Luber); we expect the same principle — smooth rather than hard boundaries between "benign" and "harmful" representations — to hold in latent space.

Training follows the self-consistent reaction field recipe, which in solvation modeling iterates the mutual polarization of solute and solvent until equilibrium (González‐Pérez et al.). We run a teacher — a reflection-style pipeline in the spirit of Wang et al., possibly a test-time reasoning model of the kind Sun et al. found navigates context naturally — over a large corpus of borderline prompts, recording its final comply-or-refuse decision. Offline, we iterate $\lambda_{t+1} = f(h - \lambda_t\,\Omega h)$ to a fixed point supervised by the teacher's decision. At inference, the head predicts the converged $\lambda$ directly in one pass: the iteration is amortized into the operator, exactly as LongCite (Zhang et al.) moved post-hoc citation checking into the generation pass itself, cutting cited snippets from roughly 220 tokens under GPT-4o to 85–90 tokens in a single forward pass. One optional extra forward pass (a fixed-point warm start) gives a latency dial between zero-cost and reflection-cost operation, letting us trace the whole accuracy–latency frontier rather than occupy one point on it.

A New Evaluation Regime: Chemistry Over-Refusal

Implicit solvation modeling supplies our application domain, because it is precisely where over-refusal blocks legitimate work. Ai et al. show that predicting whether a drug bioactivates into toxic electrophiles requires continuum dielectric calculations — gas-phase models get the barriers wrong, while aqueous and protein screening yields the documented rate-determining addition barrier of $\Delta E = -14.6$ kcal/mol for Erlotinib's side-on conformation. A toxicologist asking a model to set up exactly such a calculation is doing safety-critical, beneficial science, yet the prompt is saturated with the lexical triggers ("toxic metabolite", "reactive electrophile") that keyword-matching guardrails refuse (Röttger et al.; Ji et al.). Existing over-refusal benchmarks cover social harms and game scenarios; none covers scientific computation, and no reported result found on Lacuna for over-refusal rates on computational chemistry or solvation-workflow prompts.

We will build such a benchmark — call it SolvRefuse — of roughly 500 benign professional prompts drawn from documented solvation workflows: setting up PCM/COSMO cavities and tessellated surfaces for proteins and ions as in Steinmann et al., predicting hydration equilibrium constants in the style validated to within ±0.5 log units by González‐Pérez et al., and reasoning about bioactivation pathways like Erlotinib's. Each prompt carries sender, recipient, and purpose context fields in the format of CASE-Bench (Sun et al.), plus a contrast set of genuinely harmful chemical requests labeled by domain experts. Helpfulness is scored against the documented physical values where they exist, so we measure whether models can actually do the science when they stop refusing.

Proposed Experiments

The main evaluation runs on open-weight models (Llama-2-7B and 70B for comparability with prior numbers, plus one current open model) across XSTEST-SAFE, OR-Bench-Hard-1K, SORRY-Bench (whose 7B judge achieves over 80% Cohen's Kappa agreement with humans and evaluates 8.8K prompts in about 10 seconds on one A100, per Xie et al.), CASE-Bench, our SolvRefuse set, and safety retention on AdvBench and JailbreakBench. We report four things: benign compliance, harmful refusal, general capability (MMLU, GSM8K), and cost — extra generated tokens per query and wall-clock on fixed hardware. The decisive plots are compliance-versus-safety and compliance-versus-token-budget Pareto curves, placing our operator at zero, one, and two extra passes against reflection pipelines that spend hundreds of tokens per query. Our specific hypotheses: the single-pass operator reaches compliance within a few points of TBR's reported 0.96 on XSTEST-SAFE while generating an order of magnitude fewer tokens; it exceeds ProSafePrune's 84.5% compliance on context-dependent sets (CASE-Bench, SolvRefuse) at equal safety; and harmful refusal stays at or above ProSafePrune's 98.5% AdvBench safety score.

Baselines

The "do nothing" reference point is the unmodified model, for which the literature reports Llama-2-70B standard SFT compliance of 0.64 on XSTEST-SAFE and Llama-2-7B compliance of 49.2% on OR-Bench-Hard-1K/PHTest/OKTest (Wang et al.; Sun et al.); any intervention must beat these without collapsing safety. The strongest existing competitor is ProSafePrune (Sun et al.), which lifts LLaMA-2-7B compliance from 49.2% to 84.5% while keeping AdvBench safety at 98.5% and even raising MMLU from 37.1 to 39.6 and GSM8K from 23.0 to 25.5, finishing benchmark runs in 16 minutes where inference-time steering took 43. It is the strongest because it shares our goal — zero runtime overhead — and our diagnosis — the danger subspace. For fairness we run it on the same base checkpoints, give its rank and $\lambda$ search the same tuning budget we give our head, and evaluate both under one harness; we also test it under distribution shift, pruning on its original static sets (HEx-PHI, Alpaca, OR-Bench) and evaluating on CASE-Bench and SolvRefuse, since static prompt distributions are its stated limitation. Beating it means higher compliance on the context-dependent sets with AdvBench safety at or above 98.5% and no additional inference cost. The third comparison is TBR (Wang et al.) run at matched token budgets: at its full reflection budget it is the accuracy ceiling (0.96 compliance on XSTEST-SAFE), and at a zero-extra-token budget it cannot operate at all; no reported result found on Lacuna for TBR's decoding latency or token overhead, so we measure it ourselves. Finally, two ablations isolate our ingredients: a fixed-$\lambda$ variant ($\lambda(h)$ replaced by a tuned constant, reducing our method to an activation-level ProSafePrune) isolates input adaptivity, and an operator trained from single-round rather than iterated self-consistent distillation isolates the reaction-field training procedure.

Why not just…?

Why not just fix it with a system prompt? Röttger et al. tested exactly this: removing Llama-2.1's system prompt dropped safe-prompt refusals from 38% to 14%, but prompt tweaks are blunt — adding permissive instructions re-introduced refusals elsewhere — and Sun et al. found commercial models under sophisticated instructions still refuse nearly half of context-approved benign queries (safe context recall as low as 54.6%). The cause is geometric, not instructional: pseudo-harmful activations sit in the harmful subspace regardless of what the prompt says (Sun et al.). We will run the strongest system-prompt variant as an extra baseline and expect it to lose on SolvRefuse, where lexical density is highest.

Why not just put a guardrail model in front? Llama-Guard-3-8B scores 54.1% accuracy without context and only 60.1% with it on CASE-Bench (Sun et al.) — barely better than chance on borderline queries — and it adds a second inference hop, which is the exact multi-model latency tax the verification literature documents (Zhang et al.). An external filter cannot fix over-refusal it cannot detect; we include it as a pipeline baseline to quantify the added latency for no compliance gain.

Why not just let the model think longer? This is the real threat, and we do not argue around it: reflection works (0.96 compliance in Wang et al.), and Sun et al. observed that test-time reasoning models handle context-dependent safety without safety-specific training. Our answer is the budget-matched experiment, not an a priori claim: inference-time steering took 43 minutes where single-pass methods took 16 on identical benchmark runs (Sun et al.), and reflection spends hundreds of decoded tokens per query that interactive and batch scientific workloads cannot afford. If reflection cost ever becomes negligible, our operator loses its reason to exist — the Pareto curves we measure will show exactly where that crossover sits, and we use reasoning models as the teacher regardless, so the distillation contributes even in that world.

Why not just fine-tune on benign examples? Ji et al. measured a Spearman correlation of 0.89 between safety and over-refusal across 32 models: globally tuning a model to refuse less tends to refuse harmful things less too, and permissively tuned models like Dolphin-2.2.1 fulfilled over 50% of genuinely unsafe prompts (Xie et al.). A localized operator that only damps one subspace, with strength chosen per prompt, is our hypothesis for escaping that trade-off — but we will run direct preference optimization on the same distilled reflection data at the same compute budget, and if it matches our compliance and safety numbers, the operator was unnecessary and the contribution reduces to the data and the benchmark.

Why not just use ProSafePrune as-is? Because it cannot pass the test this proposal is built around: it applies one fixed projection regardless of context, and context is exactly what borderline safety judgments run on (Sun et al.). Our fixed-$\lambda$ ablation makes the comparison direct — if it matches the adaptive operator on CASE-Bench and SolvRefuse, the continuous-screening claim is wrong and we will say so.

Decisive Checks

Three results would kill the core claim, and the experiments are designed to detect each. If the fixed-$\lambda$ ablation matches the adaptive operator on all context-dependent benchmarks, input-dependent screening adds nothing over static pruning. If budget-matched fine-tuning matches the operator on both compliance and AdvBench safety, the mechanism is redundant to the data. And if the single-pass operator cannot get within a few points of TBR's 0.96 compliance on XSTEST-SAFE, the amortization loses too much accuracy to be worth its speed. In any of those cases, SolvRefuse itself remains a contribution: the first measurement of the refusal paradox inside scientific computing, on workflows where refusing to model a toxic molecule is itself the safety failure.