Where Does the Refusal Come From? Black-Box Attribution of the Safety Tax in Deployed AI Systems

Commercial chat and vision-language services are not single models. A user's request may pass through a hidden safety system prompt, an input classifier, the aligned model itself, and an output filter before anything comes back. When such a system refuses a harmless request, an outside auditor sees only the refusal. The auditor cannot tell which hidden stage caused it, and so cannot tell who is paying the "safety tax," meaning the benign requests lost to protection against attacks. We propose to recover that attribution from the outside. The method sends controlled pairs of queries that each stage should respond to differently, and it is calibrated on open-weight pipelines whose stage behind every refusal is known.

The payoff is diagnostic. The USB multimodal safety benchmark found that Claude 3.5 Sonnet had the lowest attack success rate of 17 models (15.57%) but refused 25.82% of harmless queries, and the authors could not say whether those refusals came from system instructions, input classifiers, the model's alignment, or output guardrails (USB). Each source implies a different fix and a different interpretation of the trade-off. An input classifier that blocks benign Swahili prompts is a replaceable component. A model whose alignment refuses "how do I kill this process" has absorbed the cost into its weights. Our aim is to turn a single aggregate over-refusal number into a per-stage account of the robustness-accuracy trade-off for systems whose internals are withheld.

Core Claim

The stages of a deployed safety pipeline differ in what they can observe and in when they act. An auditor can therefore separate them with query-level interventions, without any access to weights. Each stage has its own signature.

An input classifier sees only the user's text. Its decision should not depend on sampling temperature or on the random seed. It should not change when the auditor supplies its own system prompt. It should act before the first generated token, which shows up either in time-to-first-token or as an API-level error rather than a generated message.

A refusal produced by the model's alignment behaves differently. It varies across samples, it responds to benign clarifying context and to caller-supplied instructions, and it appears as ordinary generated tokens.

An output filter acts on what the model has generated. It can truncate a stream partway through. It changes when the maximum output length is cut, because a truncated answer may never contain the content that would trigger it.

We claim that a small set of such interventions, combined by a classifier trained on open pipelines with known stages, attributes individual refusals to stages accurately enough to decompose a black-box system's benign refusal rate. We further claim that the attribution transfers to guardrail configurations the classifier never saw. The decomposition we estimate is

$$R_{\text{benign}} = p_{\text{in}} + (1-p_{\text{in}})\,p_{\text{model}} + (1-p_{\text{in}})(1-p_{\text{model}})\,p_{\text{out}},$$

where $R_{\text{benign}}$ is the observed refusal rate on harmless queries, $p_{\text{in}}$ is the probability that the input stage blocks a request, $p_{\text{model}}$ is the probability that the model refuses a request that passed the input stage, and $p_{\text{out}}$ is the probability that the output stage suppresses a response that was produced. Computing the same terms on harmful prompts shows how much attack protection each stage buys for its share of the clean-utility cost.

We do not claim guarantees. Hair-Trigger Alignment proves that static black-box evaluation cannot separate a robust model from one with latent failure triggers. That result constrains us. A provider that deliberately randomizes its classifier or pads its latency could defeat attribution, so our identifiability rests on stated assumptions about how the stages behave. We test those assumptions directly rather than asserting them.

Relation to Prior Work

Two lines of work motivate the gap.

The first shows that hidden components shape robustness in ways outsiders cannot resolve. Wang et al. found that commercial traffic-sign recognizers resist physical attacks through an undocumented "spatial memorization" mechanism, but could not isolate its source or measure its effect on clean scenes. Prokos et al. note that cloud deployments of perceptual hashing contain defenses missing from local binaries. Visibility bias shows that client-exposed validation is only a partial blueprint of server-side logic. All three document the attribution problem. None of them builds a tool that decomposes the resulting utility cost.

The second line measures the safety tax, but only in aggregate or on isolated components. Baseline Defenses reports that a perplexity filter catches nearly all GCG suffixes (Greedy Coordinate Gradient, a discrete optimization attack) at about 10% false positives on benign queries. Yang et al. show that standalone guardrails such as MD-Judge fall from 84.62 F1 in English to 35.30 in multilingual settings, and that Aegis-Defensive becomes over-cautious in low-resource languages. ChatBug measures a model-level utility cost, with Vicuna's MT-Bench score dropping from 6.28 to 6.02 after adversarial retraining. Our work puts these components into one pipeline and asks how much of the end-to-end refusal each contributes.

Two further papers supply the method. Extracting Recurring Vulnerabilities predicts hidden backend flaws from visible UI features after profiling about 20 projects. Vote rigging on Chatbot Arena de-anonymizes models from stylistic fingerprints. We reuse their central move, which is to profile known systems and then infer hidden structure from external behavior. Where they target the identity of a model or its code flaws, we target the configuration of its defenses.

Research Program

The first component is a ground-truth testbed of assembled pipelines. We combine open aligned models (Llama-2-7B-chat, Llama-3, Mistral-7B, Vicuna, and a Zephyr-7B hardened with R2D2 from HarmBench) with interchangeable input and output stages: Llama-Guard-2, Llama-Guard-3, Aegis-Defensive, MD-Judge, and the self-perplexity filter. Each assembly can also carry an optional safety system prompt. We expose every assembly through the same API surface commercial providers offer, including streaming, temperature, a maximum-token setting, and a caller system prompt. We also add realistic latency noise. Every refusal is logged with the stage that produced it.

The second component is the probe suite. For each benign query that is refused, the auditor issues a fixed set of variants. These are resampling at several temperatures and seeds, a maximum-token setting of one token and of a short length, a caller-supplied clarifying system prompt, a benign context prefix that makes the intent explicit, a meaning-preserving paraphrase, and a translation. The auditor records the refusal outcome, the time to first token, any mid-stream truncation, and the refusal text for each variant.

The third component is the attribution model: a small classifier that maps these probe responses to a stage label. We deliberately train it on a subset of guardrails and base models and test it on held-out ones. This matters because commercial filters will not be Llama-Guard.

The fourth component is the audit itself. We apply the calibrated attributor to GPT-4o, Claude 3.5 Sonnet, and Gemini-1.5-Pro on the benign and harmful splits of USB and on multilingual benign prompts. The output is a per-stage version of the decomposition above for each system and for each language.

Baselines

The naive reference point is aggregate measurement: reporting only $R_{\text{benign}}$ and attack success rate per system, as USB does. Its reported figures are 25.82% oversensitivity and 15.57% ASR for Claude 3.5 Sonnet. It attributes nothing, so in our metric it amounts to assigning every refusal to a single "system" label. We rerun the USB protocol on the same query set so that our aggregate numbers are checked against theirs before we decompose them.

The strongest cheap competitor is refusal-text matching. This heuristic labels canned or templated refusals and API errors as filter refusals, and fluent explanatory refusals as model refusals. Providers that return fixed moderation messages may make this nearly sufficient. It is the baseline we most need to beat. We run it on identical logs, with its templates fit on the same training pipelines. Beating it means higher macro-F1 across the three stages on held-out configurations, especially when we randomize the refusal text of filter stages in the testbed. No reported result was found on Lacuna for black-box stage attribution of refusals by any method.

We also compare against a timing-only side channel and against model self-report, in which the system is asked why it refused.

The ablation that isolates our key ingredient removes the interventional probes, meaning resampling, the caller system prompt, and maximum-token manipulation, while keeping passive signals such as text, latency, and truncation. If the interventions do not add accuracy beyond the passive signals, the core claim fails.

All methods receive the same query budget per refused prompt. The budget is capped at the number of calls the full probe suite uses, so that fairness reflects real audit cost.

Why not just…?

Why not just read the refusal message? On some providers this will work, because a fixed moderation string reveals a filter. That is exactly why it is the main baseline and not a straw man. It breaks in three cases: when the filter hands off to the model to phrase the refusal, when an output filter truncates silently, and when a model's alignment produces formulaic refusals. The randomized-template condition in the testbed measures how much attribution survives once text cues are removed. If text matching matches our accuracy there, we will report that the cheap method suffices.

Why not just compare the API against the open-weight version of the same model? This works only where an open-weight twin exists, and it fails for the systems that matter most. Prokos et al. already show that local replicas omit cloud-side defenses, so the gap between twin and API conflates all hidden stages into one residual. We include this comparison for Llama-hosted endpoints as a sanity check on our input-stage estimates.

Why not just use timing? Latency is noisy across load and regions, and a provider can pad it cheaply. The timing-only baseline under injected jitter shows how fragile it is. Our expectation is that timing helps separate input-stage refusals but cannot distinguish model refusals from output-filter refusals. Maximum-token and resampling probes do make that distinction.

Why not just ask the model? Model self-reports are generated text with no privileged view of upstream filters, so a model will tend to rationalize a refusal it never made. The self-report baseline quantifies this on the testbed, where the true stage is known.

Proposed Experiments and Decisive Checks

On the testbed, the primary metric is macro-F1 of stage attribution on held-out guardrail and base-model combinations. The secondary metric is absolute error in the estimated $p_{\text{in}}$, $p_{\text{model}}$, and $p_{\text{out}}$ relative to logged ground truth. Benign queries come from USB and the multilingual guardrail datasets. Harmful queries come from HarmBench and AdvBench, and MT-Bench tracks general utility.

As a calibration check, the perplexity-filter stage in the testbed should reproduce roughly the 10% benign false-positive rate reported in Baseline Defenses. If it does not, our benign set differs from theirs, and we will say so.

For commercial systems, where ground truth is unavailable, the decisive test is interventional prediction. For every refusal the attributor assigns to the model stage, we predict that a clarifying caller system prompt will reverse a substantial fraction of refusals. For every refusal assigned to the input stage, we predict that the same prompt will reverse almost none, while a meaning-preserving translation or paraphrase that avoids the filter's surface cues will. We fit these predictions on held-out queries and pre-register them before running.

If the attributor's labels do not predict which repair works, then it is not recovering real structure, whatever its testbed accuracy. If they do, we will be able to report, for the first time, how much of a proprietary system's benign refusal rate, in English and in low-resource languages, is imposed by wrappers that could be replaced rather than by the model itself. We will query commercial APIs within their rate limits and terms of service, and every probe query will be benign or drawn from existing public harmful-behavior benchmarks.