Abstract
Large language models (LLMs) have demonstrated immense utility across various industries. However, as LLMs advance, the risk of harmful outputs increases due to incorrect or malicious instruction prompts. While current methods effectively address jailbreak risks, they share common limitations: 1) Judging harmful responses from the prefill-level lacks utilization of the model's decoding outputs, leading to relatively lower effectiveness and robustness. 2) Rejecting potentially harmful responses based on a single evaluation can significantly impair the model's helpfulness. This paper examines the LLMs' capability to recognize harmful outputs, revealing and quantifying their proficiency in assessing the danger of previous tokens. Motivated by pilot experiment results, we design a robust defense mechanism at the decoding level. Our novel decoder-oriented, step-by-step defense architecture corrects harmful queries directly rather than rejecting them outright. We introduce speculative decoding to enhance usability and facilitate deployment to boost secure decoding speed. Extensive experiments demonstrate that our approach improves model security without compromising reasoning speed. Notably, our method leverages the model's ability to discern hazardous information, maintaining its helpfulness compared to existing methods.
Introduction
Current Large Language Model (LLM) safety measures generally fall into two categories: prefill-level (checking the user's prompt) and output-level (checking the final response). Both approaches possess significant flaws. Prefill defenses are often bypassed by sophisticated "jailbreak" prompts that obscure malicious intent. Conversely, output-level defenses are "all-or-nothing"—they frequently reject entire benign responses due to a single misinterpreted word and introduce significant latency because the model must complete generation before filtering begins.
Researchers have identified a "root" opportunity: LLMs exhibit internal signals indicating when they are generating harmful content. The challenge lies in harnessing this internal state to correct the model during the decoding process without sacrificing speed or utility.
Methodology: Root Defence Strategies (RDS)
The RDS framework introduces a novel two-pronged architecture designed to intervene at the token-by-token decoding level.
- Step-by-Step Safe Generation: Instead of a binary filter, RDS utilizes a lightweight, trainable classifier that monitors the model’s hidden states at every decoding step. It evaluates the "harmfulness score" of the top-k candidate tokens. If a token is deemed risky, RDS adjusts the logits to prioritize safer, benign alternatives. This allows the model to pivot from a harmful start to a safe disclaimer mid-sentence.
- Speculative Decoding: To offset the computational cost of per-step classification, RDS integrates a speculative decoding head (specifically, an "EAGLE head"). This allows the model to predict future hidden states and tokens in parallel, significantly boosting generation speed by up to 3x.
Experimental Setup
The researchers evaluated RDS across diverse benchmarks to test both safety and helpfulness:
- Safety Training: A custom dataset derived from DRO (100 harmful, 100 benign queries).
- Harmful Benchmarks: HEx-PHI, AdvBench, and MaliciousInstruct to test resistance to jailbreaks.
- Benign Benchmarks: Held-out and Xstest (designed to trigger "false positive" refusals) to measure over-defense.
- Utility Evaluation: The Just-Eval dataset, with GPT-4 scoring the quality of helpful responses.
Results and Insights
Internal Discernment
A key finding is that LLMs naturally differentiate between harmful and benign tokens in their high-dimensional hidden states. Principal Component Analysis (PCA) reveals how harmful tokens cluster away from benign ones as early as the first few tokens of a response.
The authors also observed that safety is a gradient; for harmful queries, even "benign" tokens (like a refusal message) are closer to the "harmful" side of the latent space than tokens in a truly benign query.
Performance and Safety
- Superior Safety: On Vicuna-7B, RDS reduced the compliance rate with harmful queries from 42.3% to just 5.3%.
- Lower Over-Defense: In the Xstest benchmark, RDS maintained a much lower refusal rate compared to methods like Self-Examination, which often blocked benign queries entirely.
- Efficiency: Thanks to speculative decoding, RDS achieved token generation speeds 2.12x to 3.09x faster than baseline models.
- Correction over Rejection: By intervening at the logit level, RDS "nudges" the model to provide safe explanations instead of outright rejection, preserving the model's overall helpfulness.
Conclusion
RDS demonstrates that intervening at the decoding level is a robust and efficient way to secure LLMs. While the system occasionally triggers false refusals on "subconscious" harm (e.g., technical terms like "killing a process"), it significantly outperforms existing prefill and output-level defenses. Future work will focus on further optimizing the model's overcorrection tendencies to ensure maximum helpfulness in "gray area" queries.
Summary updated