System Prompts Affect Response Bias More Than Discriminability in LLM Safety

Mohan W. Gupta, Sandy TanwisutharXiv 2026 · arXiv

TL;DR

System prompts don't necessarily make Large Language Models (LLMs) "smarter" at identifying unsafe requests; instead, they primarily adjust the model's "trigger finger" for refusal. By applying Signal Detection Theory, researchers found that while a model's fundamental ability to distinguish safe from unsafe queries is tied to its identity, system prompts and instructions mainly shift the response bias—making models more or less cautious without improving their underlying safety judgment.

What Problem Does This Paper Address?

LLM safety is typically evaluated as a binary tradeoff between compliance and refusal: we want models to refuse harmful prompts and comply with safe ones. However, a change in refusal rates can be deceptive. Does a model refuse more because it has become better at detecting harm, or simply because it has become more "trigger-happy"?

Furthermore, the authors argue that safety is not just a binary choice. A model might respond to a sensitive query by asking for clarification, providing "safe help" (benign alternatives), or isolating untrusted data. This paper seeks to understand how system prompts route models into these different behavioral policies and whether these prompts actually improve a model's ability to discriminate between safe and unsafe content.

Methods

The researchers employed a controlled evaluation framework using Signal Detection Theory (SDT) to decouple two distinct factors:

  1. Discriminability ($d'$): The model's fundamental ability to tell the difference between a "signal" (unsafe request) and "noise" (safe request).
  2. Response Criterion ($c$): The model's internal threshold or bias toward refusing, regardless of its accuracy.

They tested 13 different models (including Llama 3.1, Qwen 2.5, and frontier models like GPT-6 Astra and Claude 5.5) by crossing eight prompt families (e.g., "Refuse-first," "Clarify-first," "Evidence-first") with three clarity levels (Vague, Explicit, and Explicit with Fallback).

To move beyond binary scoring, they developed a taxonomy of six output policies: Compliance, Refusal, Clarification, Hierarchy Preservation, Safe Help, and Source Isolation. A blind LLM-based judge was used to classify the models' actual responses into these categories.

The experimental design: routing a fixed user query through different system prompts to observe changes in behavioral policy.

Datasets

The study utilized 3,370 items drawn from three major benchmarks:

  • HarmBench: Focuses on harmful requests where models should refuse or provide safe help.
  • XSTest: Contains safe but sensitive "check" items to test for over-refusal, alongside unsafe contrast items.
  • IHEval: Tests instruction-following and the ability to prioritize system instructions over conflicting user prompts (hierarchy preservation and source isolation).

Key Insights and Ideas

  • Model Identity is King for $d'$: A model's ability to distinguish safe from unsafe is largely inherent to the model itself. If a model is fundamentally "confused" by a specific type of harm, a system prompt telling it to "be safe" won't fix that confusion; it will just make the model refuse more safe prompts along with the unsafe ones.
  • Prompts are Bias Shifters: System prompts primarily act on the response criterion ($c$). For example, moving from an "Answer" prompt to a "Refuse" prompt shifts the model's threshold, making it more likely to refuse across the board without significantly changing its accuracy in doing so.
  • Clarity Increases Caution, Not Skill: Increasing the clarity of a safety prompt (e.g., providing explicit fallback rules) makes models more willing to refuse, but it does not reliably improve their ability to distinguish safe from unsafe requests.
  • Safety as Task Selection: The authors show that system prompts effectively "route" the model into different behavioral modes. A "Clarify" prompt doesn't just change refusal rates; it significantly increases the probability that a model will ask a follow-up question rather than acting prematurely.

Key Results

  • Discriminability vs. Bias: Model identity was a massive predictor of discriminability ($F=258.0, p < .001$), while prompt family had a much larger and more consistent effect on the response criterion ($F=16.61, p < .001$).
  • ROC Analysis: As shown in Figure 2, different system prompts (colored dots) generally move along the same Receiver Operating Characteristic (ROC) curve for a given model. This indicates that the prompts are changing the operating point (bias) rather than shifting the curve itself (discriminability).
Figure 2: Each panel shows a model’s refusal hit rate on unsafe requests against its false-alarm rate on safe requests. Colored points represent prompt families, aggregated across clarity levels. Black curves show the theoretical ROC implied by each model’s mean discriminability (d′), reported above each panel. Dashed diagonal lines indicate chance discrimination.
  • Policy Routing: Prompt families successfully biased models toward their intended behaviors. For instance, Hierarchy prompts increased hierarchy preservation behavior from 1.32% to 9.12% (an odds ratio of 7.48). Clarify prompts increased clarification from 32.3% to 47.4%.
Figure 3: Prompt families bias models toward differ- ent behavioral policies. Each row shows a prompt fam- ily, and the x-axis shows the percentage of responses assigned to each of the six output policies. Colors in- dicate the observed policy. Prompt families generally increased their corresponding target behaviors. Bars show the unweighted mean of model-specific response proportions, giving each model equal weight. Inferen- tial analyses use pooled response counts.

Limitations and Future Work

  • Judge Agreement: The authors noted that identifying "Safe Help" is difficult even for human raters and LLM judges, leading to lower agreement scores for that specific category.
  • Output-Only Judging: The judge only saw the model's response, not the original prompt. While this prevents the judge from being biased by the prompt's intent, it means the judge cannot always determine if a response was correctly routed, only what behavior it displayed.
  • Real-World Complexity: Deployed systems often use complex, multi-objective system prompts. This study isolated individual safety priorities; future work could explore how these effects interact when multiple safety rules are combined in a single prompt.
  • Mechanistic Interpretability: The findings provide a behavioral map that could guide future research into the internal "task representations" models form when processing system prompts.

Summary updated

Abstract

Large language model safety evaluations often focus on a compliance-refusal tradeoff: models should refuse unsafe requests without refusing safe ones. We argue that this misses a broader form of prompt-conditioned task selection, where the same request can elicit different behaviors depending on the system prompt. We investigate this using a controlled evaluation crossing eight safety-task prompt families with three clarity levels across HarmBench, XSTest, and IHEval. Holding benchmark items fixed, we evaluate 13 models and use signal detection theory to distinguish two possibilities: whether models become better at telling safe from unsafe requests (discriminability), or simply more willing to refuse (response criterion). We find substantial differences in discriminability across models, but system prompts primarily shift the willingness to refuse, with smaller and less consistent effects on discriminability. Increasing prompt clarity makes models more willing to refuse without reliably improving their ability to distinguish safe from unsafe requests. Beyond refusal, prompt families also shift responses among six distinct behavioral policies, including clarification and safe help. Together, these findings show that system prompts primarily change how models respond to safety-relevant requests rather than consistently improving their ability to distinguish safe from unsafe ones.
BibTeX
@misc{gupta2026system,
  author = {Mohan W. Gupta and Sandy Tanwisuth},
  title = {{System Prompts Affect Response Bias More Than Discriminability in LLM Safety}},
  year = {2026},
  eprint = {2611.11074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11074},
}
Figure