PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing

Blazej Manczak, Eliott Zemour, Eric Lin, Vaikkunth Mugunthan

TL;DR

PrimeGuard is a novel, tuning-free inference-time guardrailing method that eliminates the "guardrail tax"—the frustrating trade-off where making an LLM safer makes it less helpful. By using structured control flow to route user queries through different self-instantiations of the same language model, PrimeGuard simultaneously boosts safety compliance to 97% and improves helpfulness scores, all while drastically reducing jailbreak vulnerability.

What Problem Does This Paper Address?

When deploying Large Language Models (LLMs), developers face a dual challenge: models must be helpful and high-quality, yet strictly compliant with safety guidelines. Standard approaches like Reinforcement Learning from Human Feedback (RLHF) align models generally but struggle with custom, deployment-specific safety policies.

To enforce these policies, developers turn to Inference-Time Guardrails (ITG). However, existing ITG methods suffer from a guardrail tax (analogous to the alignment tax): if you tune the guardrail to be highly secure, the model becomes overly cautious and refuses benign prompts (high false-refusal rate). If you prioritize helpfulness, the model becomes vulnerable to adversarial jailbreaks. This paper introduces a routing mechanism designed to break this Pareto frontier.

Methods

PrimeGuard is a tuning-free approach that leverages the inherent instruction-following and in-context learning (ICL) capabilities of the base LLM. Instead of training a separate classifier, it instantiates two roles from the same underlying model: $LLM_{Guard}$ and $LLM_{Main}$.

The system operates in two distinct stages:

  1. Stage 1: Risk-Aware Routing: The user query is sent to $LLM_{Guard}$, which has access to both directive instructions (what to do) and restrictive instructions (what not to do). It categorizes the query into one of three risk levels: No Violation, Direct Violation, or Potential Violation. It also generates a brief rationale and a query-specific "guidance tip."
  2. Stage 2: Response Generation: The query is routed based on Stage 1's classification:
  3. No Violation: Routed to $LLM_{Main}$ with instructions to maximize helpfulness using the dynamic guidance tip.
  4. Direct Violation: Routed to $LLM_{Main}$ with instructions to generate a polite refusal.
  5. Potential Violation: Routed back to $LLM_{Guard}$ for a deeper, nuanced re-evaluation to handle borderline or stealthy jailbreak queries safely.
Figure 1. PrimeGuard approach overview. The routes maximize adherence to directives (do’s) and restrictives (don’ts).

To ensure high performance without fine-tuning, the authors systematically generate synthetic ICL examples that teach $LLM_{Guard}$ how to classify risks and use specific stylistic tokens (like "I apologize" or "Hello! Great question") to shape the model's output distribution.

Datasets

The authors evaluated PrimeGuard using a mix of standard benchmarks and a newly curated dataset:

  • safe-eval (New): A diverse red-teaming benchmark consisting of 1,741 unsafe prompts compiled from existing datasets (AART, Student-Teacher-Prompting, SAP, Jailbreakbench, and XSTest).
  • HardTapPrompts (New subset of safe-eval): 614 prompts that successfully jailbroke advanced models like GPT-4 and GPT-3.5 using the Tree of Attacks (TAP) method.
  • XSTest: Used specifically to evaluate over-sensitivity and measure false refusal rates on benign prompts that look superficially risky.
  • Dolly-15k: 200 sampled prompts used to evaluate the model's helpfulness across 8 behavioral categories.

Key Insights and Ideas

  • The Power of Re-Evaluation: Borderline prompts are notoriously difficult for static guardrails. PrimeGuard’s "Potential Violation" route acts as an exception-handling routine, giving the model a second look at nuanced risks.
  • ICL Specialization: You don't need to fine-tune a guardrail model. By carefully structuring in-context examples, a single base model can effectively police itself.
  • Complementary ICL Effects: The ablation studies revealed an interesting division of labor: adding more routing examples drastically improves safety, whereas adding more re-evaluation examples primarily boosts helpfulness on borderline queries.
Figure 3. Adding ICL examples during routing increases safety while examples during re-evaluation increase helpfulness.

Key Results

PrimeGuard significantly outperformed state-of-the-art baselines (like Self-Reminder and Intention-Analysis) across multiple model sizes (Mixtral-8x22B, Mistral-7B, and Phi-3-mini):

  • Breaking the Guardrail Tax: On Mixtral-8x22B, PrimeGuard increased the fraction of safe responses from 60.8% (alignment only) to 97.0%, while actually increasing the average helpfulness score from 4.17 to 4.29.
  • Jailbreak Resilience: Against the state-of-the-art automated jailbreak attack (TAP), PrimeGuard reduced the Attack Success Rate (ASR) on Mixtral-8x22B from 100% down to just 8%, outperforming even stringently aligned models like Llama-3-8B.
Figure 2. For each model, PrimeGuard achieves high helpfulness-safety scores than alternatives, including RLHF alignment.
Table 1. Iterative Attack Success Rates (ASR) and average number of queries to jailbreak (#Queries) for successful TAP attacks.
  • Zero False Refusals: For the large Mixtral model, PrimeGuard brought the false refusal rate on benign prompts down to 0.0%.

Limitations and Future Work

  • Scale Dependence: PrimeGuard relies heavily on the model's inherent instruction-following capabilities. The authors noted that the benefits decrease significantly for models smaller than 7B parameters.
  • Structured Output Failures: Because Stage 1 requires the model to output a structured JSON dictionary, there is a 1% to 5% failure rate where the model generates invalid syntax. The authors suggest that integrating logit-processing constraints (like Outlines or Guidance) could resolve this.
  • Future Directions: Future work will focus on deriving higher levels of helpfulness from smaller models and expanding the safe-eval benchmark to cover more evolving adversarial attack vectors.

Summary updated

Abstract

Deploying language models (LMs) necessitates outputs to be both high-quality and compliant with safety guidelines. Although Inference-Time Guardrails (ITG) offer solutions that shift model output distributions towards compliance, we find that current methods struggle in balancing safety with helpfulness. ITG Methods that safely address non-compliant queries exhibit lower helpfulness while those that prioritize helpfulness compromise on safety. We refer to this trade-off as the guardrail tax, analogous to the alignment tax. To address this, we propose PrimeGuard, a novel ITG method that utilizes structured control flow. PrimeGuard routes requests to different self-instantiations of the LM with varying instructions, leveraging its inherent instruction-following capabilities and in-context learning. Our tuning-free approach dynamically compiles system-designer guidelines for each query. We construct and release safe-eval, a diverse red-team safety benchmark. Extensive evaluations demonstrate that PrimeGuard, without fine-tuning, overcomes the guardrail tax by (1) significantly increasing resistance to iterative jailbreak attacks and (2) achieving state-of-the-art results in safety guardrailing while (3) matching helpfulness scores of alignment-tuned models. Extensive evaluations demonstrate that PrimeGuard, without fine-tuning, outperforms all competing baselines and overcomes the guardrail tax by improving the fraction of safe responses from 61% to 97% and increasing average helpfulness scores from 4.17 to 4.29 on the largest models, while reducing attack success rate from 100% to 8%. PrimeGuard implementation is available at https://github.com/dynamofl/PrimeGuard and safe-eval dataset is available at https://huggingface.co/datasets/dynamoai/safe_eval.
BibTeX
@inproceedings{manczak2024primeguard,
  author = {Blazej Manczak and Eliott Zemour and Eric Lin and Vaikkunth Mugunthan},
  title = {{PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing}},
  booktitle = {NextGenAISafety @ ICML},
  year = {2024},
  url = {https://openreview.net/forum?id=GcG3zVkie2},
}
Figure