TL;DR
System prompts don't necessarily make Large Language Models (LLMs) "smarter" at identifying unsafe requests; instead, they primarily adjust the model's "trigger finger" for refusal. By applying Signal Detection Theory, researchers found that while a model's fundamental ability to distinguish safe from unsafe queries is tied to its identity, system prompts and instructions mainly shift the response bias—making models more or less cautious without improving their underlying safety judgment.
What Problem Does This Paper Address?
LLM safety is typically evaluated as a binary tradeoff between compliance and refusal: we want models to refuse harmful prompts and comply with safe ones. However, a change in refusal rates can be deceptive. Does a model refuse more because it has become better at detecting harm, or simply because it has become more "trigger-happy"?
Furthermore, the authors argue that safety is not just a binary choice. A model might respond to a sensitive query by asking for clarification, providing "safe help" (benign alternatives), or isolating untrusted data. This paper seeks to understand how system prompts route models into these different behavioral policies and whether these prompts actually improve a model's ability to discriminate between safe and unsafe content.
Methods
The researchers employed a controlled evaluation framework using Signal Detection Theory (SDT) to decouple two distinct factors:
- Discriminability ($d'$): The model's fundamental ability to tell the difference between a "signal" (unsafe request) and "noise" (safe request).
- Response Criterion ($c$): The model's internal threshold or bias toward refusing, regardless of its accuracy.
They tested 13 different models (including Llama 3.1, Qwen 2.5, and frontier models like GPT-6 Astra and Claude 5.5) by crossing eight prompt families (e.g., "Refuse-first," "Clarify-first," "Evidence-first") with three clarity levels (Vague, Explicit, and Explicit with Fallback).
To move beyond binary scoring, they developed a taxonomy of six output policies: Compliance, Refusal, Clarification, Hierarchy Preservation, Safe Help, and Source Isolation. A blind LLM-based judge was used to classify the models' actual responses into these categories.
Datasets
The study utilized 3,370 items drawn from three major benchmarks:
- HarmBench: Focuses on harmful requests where models should refuse or provide safe help.
- XSTest: Contains safe but sensitive "check" items to test for over-refusal, alongside unsafe contrast items.
- IHEval: Tests instruction-following and the ability to prioritize system instructions over conflicting user prompts (hierarchy preservation and source isolation).
Key Insights and Ideas
- Model Identity is King for $d'$: A model's ability to distinguish safe from unsafe is largely inherent to the model itself. If a model is fundamentally "confused" by a specific type of harm, a system prompt telling it to "be safe" won't fix that confusion; it will just make the model refuse more safe prompts along with the unsafe ones.
- Prompts are Bias Shifters: System prompts primarily act on the response criterion ($c$). For example, moving from an "Answer" prompt to a "Refuse" prompt shifts the model's threshold, making it more likely to refuse across the board without significantly changing its accuracy in doing so.
- Clarity Increases Caution, Not Skill: Increasing the clarity of a safety prompt (e.g., providing explicit fallback rules) makes models more willing to refuse, but it does not reliably improve their ability to distinguish safe from unsafe requests.
- Safety as Task Selection: The authors show that system prompts effectively "route" the model into different behavioral modes. A "Clarify" prompt doesn't just change refusal rates; it significantly increases the probability that a model will ask a follow-up question rather than acting prematurely.
Key Results
- Discriminability vs. Bias: Model identity was a massive predictor of discriminability ($F=258.0, p < .001$), while prompt family had a much larger and more consistent effect on the response criterion ($F=16.61, p < .001$).
- ROC Analysis: As shown in Figure 2, different system prompts (colored dots) generally move along the same Receiver Operating Characteristic (ROC) curve for a given model. This indicates that the prompts are changing the operating point (bias) rather than shifting the curve itself (discriminability).
- Policy Routing: Prompt families successfully biased models toward their intended behaviors. For instance, Hierarchy prompts increased hierarchy preservation behavior from 1.32% to 9.12% (an odds ratio of 7.48). Clarify prompts increased clarification from 32.3% to 47.4%.
Limitations and Future Work
- Judge Agreement: The authors noted that identifying "Safe Help" is difficult even for human raters and LLM judges, leading to lower agreement scores for that specific category.
- Output-Only Judging: The judge only saw the model's response, not the original prompt. While this prevents the judge from being biased by the prompt's intent, it means the judge cannot always determine if a response was correctly routed, only what behavior it displayed.
- Real-World Complexity: Deployed systems often use complex, multi-objective system prompts. This study isolated individual safety priorities; future work could explore how these effects interact when multiple safety rules are combined in a single prompt.
- Mechanistic Interpretability: The findings provide a behavioral map that could guide future research into the internal "task representations" models form when processing system prompts.
Summary updated