Ensuring the safety of Large Language Models (LLMs) has traditionally relied on static training methods like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). However, these defenses are often fragile, susceptible to jailbreaking via adversarial prompts, and can be inadvertently compromised during model editing. Presented at AAAI-25, SafeInfer introduces a novel two-stage, "plug-and-play" defense mechanism that operates during inference to maintain safety without sacrificing general utility.
The Challenge of Static Alignment
Traditional safety alignment faces two primary hurdles. First, jailbreaking techniques—ranging from complex "Chain of Thought" prompts to gradient-based attacks—can trick models into bypassing their safety filters. Second, model editing (e.g., using ROME to update knowledge) often breaks established safety guardrails. SafeInfer addresses these by shifting the defense to the decoding stage, allowing for real-time, context-adaptive moderation that avoids the high costs of retraining.
Methodology: A Two-Stage Defense
SafeInfer employs a "judicious coupling" of internal activation steering and output distribution filtering. This dual-layered approach ensures that safety is addressed both in the model's latent representations and its final token selection.
Phase 1: Safety Amplification (SA)
This phase targets the model's internal latent space. By analyzing how attention heads behave during safe demonstrations, the authors derive a Safety Amplification Vector (SV). During a user query's forward pass, this SV is added to the hidden states at a specific layer (found to be most effective at the 9th layer for Llama-2). This intervention "nudges" the model toward a safer internal state before token generation begins.
Phase 2: Safety Guided Decoding Strategy (sGDS)
This phase filters the output distribution. SafeInfer utilizes a small, intentionally harmful reference model ($M_{unsafe}$). During decoding, the system compares the target model's probability distribution against the harmful model's. Tokens that the harmful model is likely to select (e.g., the start of a prohibited instruction) are suppressed, biasing the final selection toward safe, helpful responses.
Evaluation and Benchmarking
The authors introduced HarmEval, a comprehensive benchmark featuring ~550 queries across 11 categories—including malware, child abuse, and illegal activity—aligned with the usage policies of major AI developers. SafeInfer was also tested against standard benchmarks like DangerousQA and AdvBench, as well as utility tests like MMLU and TruthfulQA to monitor for "over-refusal."
Key Results and Insights
SafeInfer demonstrates a significant reduction in vulnerability across multiple architectures, including Llama-2, Mistral, Vicuna, and Falcon. On Llama-2, the Attack Success Rate (ASR) on HarmEval dropped from 21.63% to 1.09%. Against gradient-based GCG attacks, the success rate fell from 0.37 to 0.07.
Crucially, SafeInfer maintains the model's core intelligence. Scores on MMLU and GSM8K remained nearly identical to base models, proving that safety does not have to come at the cost of capability. Furthermore, the system successfully "re-aligned" models whose safety had been compromised by knowledge editing, highlighting its utility for long-term model maintenance.
Conclusion and Future Directions
SafeInfer proves that context-adaptive decoding is a powerful tool for LLM safety. While the system requires careful hyperparameter tuning (specifically the $\gamma$ and $\lambda$ values) and access to a harmful reference model, its ability to suppress harmful outputs while preserving utility is a major step forward. Future work may extend this approach to multi-modal models and explore the scaling properties of safety vectors in even larger architectures.
Summary updated