Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs

Yue Huang, Haomin Zhuang, Jiayi Ye, Han Bao, Yanbo Wang, Hang Hua, Siyuan Wu, Pin-Yu Chen, Xiangliang Zhang

TL;DR

Traditional LLM safety guardrails rely on "hard gating" to block risky queries, which often leads to frustrating over-refusal of benign requests. This paper introduces Guardian-as-an-Advisor (GaaA), a soft-gating framework that prepends constructive safety and trustworthiness advice directly to user prompts, steering the base model to fulfill nuanced requests safely without flat-out blocking them.

What Problem Does This Paper Address?

Current large language model (LLM) guardrails typically act as rigid binary filters. If a user query looks remotely risky, the guardrail triggers a "hard block," replacing the model's response with a generic refusal template like "Sorry, I cannot help with that."

This approach suffers from two major flaws:

  1. Over-refusal: It frequently blocks completely benign or policy-compliant creative and educational requests (e.g., discussing reproductive health or writing a dramatic story with romantic tension).
  2. Narrow Scope: Standard safety taxonomies focus almost exclusively on malicious intent (toxicity, jailbreaks), completely ignoring other critical pillars of trustworthiness like robustness (handling typos or noisy text without tripping safety alarms) and honesty (knowing model limitations).

This creates a false trade-off where AI systems are safer on paper but significantly less useful in practice.

Comparison of traditional hard-gating guardrails vs. the proposed Guardian-as-an-Advisor soft-gating paradigm.

Methods

The authors propose Guardian-as-an-Advisor (GaaA), a soft-gating pipeline. Instead of blocking a query, a lightweight guardian model (GuardAdvisor) evaluates the input and generates a binary risk label along with a natural-language explanation. This advice is prepended directly to the original user query (i.e., [Risk = c; Explanation = e] || x), allowing the downstream model to self-regulate and respond safely within its original specification.

To train GuardAdvisor (built on Qwen-2.5-7B-Instruct), they use a two-stage recipe:

  1. Supervised Fine-Tuning (SFT): Teaching the model to generate structured labels and explanations based on a new dataset.
  2. Reinforcement Learning (RL): Utilizing Group-Relative Policy Optimization (GRPO) with an LLM-as-a-Judge reward function. This step is crucial to calibrate the policy and prevent "reward hacking," ensuring the generated explanation semantically aligns with the predicted label.

To minimize deployment latency, they introduce a Parallel Inference Strategy. GuardAdvisor evaluates the query simultaneously as the base model begins its first inference pass. If a risk is detected, the base model's generation is interrupted and restarted with the advisor's guidance; otherwise, it proceeds without interruption.

Parallel inference strategy to minimize latency overhead.

Datasets

The authors introduce GuardSet, a large-scale, multi-domain dataset containing over 208,000 data points. GuardSet unifies 55+ public datasets covering both harmful and harmless scenarios. Crucially, it extends harmless data with curated slices specifically targeting robustness (adding natural noise like typos and slang) and honesty (recognizing epistemic boundaries and limitations).

The data pipeline consists of three stages: collection, processing (label mapping and DeepSeek-R1 explanation synthesis), and validation (GPT-4o-mini filtering combined with human spot-checks).

Figure 2: GuardSet construction pipeline.

Key Insights and Ideas

  • Binary Labels + Rich Explanations: Instead of relying on brittle, fine-grained taxonomies where overlapping risks cause classification failures, GaaA simplifies the top-level classification to Harmful/Harmless and offloads the nuance to the natural-language explanation.
  • Trustworthiness Beyond Safety: Trust isn't just about avoiding harm; it's also about how a model handles noisy inputs (robustness) and whether it knows what it doesn't know (honesty).
  • The SFT Over-Caution Trap: The authors found that scaling up SFT data alone makes the guardian model overly cautious, increasing over-refusal on benign inputs. RL via GRPO is essential to calibrate the model's judgment.
  • Amortizing Latency via Parallelism: Because harmful queries are statistically rare in real-world production, running the guardrail in parallel and only interrupting on true risks keeps the average latency overhead negligible.

Key Results

  • High Accuracy: GuardAdvisor achieves an average detection accuracy of 90.52%, outperforming GPT-4o-mini (88.16%) and performing competitively with GPT-4o (91.40%) despite handling a more complex, fine-grained explanation task.
  • Improved Response Quality: Prepending GuardAdvisor's explanations significantly steers the base model to perform better. On robustness tasks, it improves the base model's win rate to 63.11% (outperforming GPT-4o's 39.48%). On honesty tasks, it hits a 68.79% win rate.
  • Minimal Latency Overhead: GuardAdvisor is highly efficient, requiring less than 5% of the base model's compute. Under realistic harmful input rates, the end-to-end latency overhead added by GaaA is only 2% to 10%.
Impact of SFT data size on accuracy and inference time comparisons.

Limitations and Future Work

  • Evolving Real-World Risks: The static dataset and evaluation protocol cannot fully capture the open-ended, rapidly evolving nature of real-world adversarial attacks and jailbreaks.
  • Proxy-Based Guarantees: The theoretical safety guarantees rely on approximate empirical modeling and LLM-as-a-judge proxies, which may miss subtle real-world deployment nuances.
  • Broader Trust Challenges: The current system does not explicitly address adaptive misuse, severe distributional drift, or complex fairness considerations, which remain open areas for future research.

Summary updated

Abstract

Hard-gated safety checkers often over-refuse and misalign with a vendor's model spec; prevailing taxonomies also neglect robustness and honesty, yielding safer-on-paper yet less useful systems. This work introduces Guardian-as-an-Advisor (GaaA), a soft-gating pipeline where a guardian predicts a binary risk label plus a concise explanation and prepends this advice to the original query for re-inference, keeping the base model operating under its original spec. To support training and evaluation, GuardSet is constructed, a 208k+ multi-domain dataset unifying harmful and harmless cases with targeted robustness and honesty slices. GuardAdvisor is trained via SFT followed by RL to enforce label-explanation consistency. GuardAdvisor attains competitive detection accuracy while enabling the advisory workflow; when used to augment inputs, responses improve over unaugmented prompts. A latency study shows advisor inference uses below 5% of base-model compute and adds only 2-10% end-to-end overhead under realistic harmful-input rates. Overall, GaaA steers models to comply with the model spec, maintaining safety while reducing over-refusal.
BibTeX
@inproceedings{huang2026guardian,
  author = {Yue Huang and Haomin Zhuang and Jiayi Ye and Han Bao and Yanbo Wang and Hang Hua and Siyuan Wu and Pin-Yu Chen and Xiangliang Zhang},
  title = {{Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.292},
}
Figure