GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners

Haoran Li, Yulin Chen, Jingru Zeng, Hao Peng, Huihao Jing, Wenbin Hu, Xi Yang, Ziqian Zeng, Sirui Han, Yangqiu SongSubmission to ICLR 2026 · Withdrawn

TL;DR

Existing LLM guardrails are typically rigid—either limited to coarse safe/unsafe binary labels or locked into a single benchmark’s fixed set of safety categories. GSPR (Generalizable Safety Policy Reasoner) turns safety policies into dynamic prompt variables and trains the model via Group Relative Policy Optimization (GRPO) to output both binary judgments and fine-grained policy violations, achieving state-of-the-art moderation performance with concise reasoning traces.

What Problem Does This Paper Address?

As LLMs are deployed across diverse domains, external "guardrail" models (like LLaMA Guard, ShieldGemma, or GuardReasoner) act as safety firewalls to moderate user prompts and model responses. However, current guardrails face critical bottlenecks:

  1. Coarse-Grained Judgments: Many models only predict binary safe/unsafe labels, ignoring fine-grained categories (e.g., distinguishing hate speech from copyright infringement or self-harm).
  2. Fixed, Inflexible Taxonomies: Existing guardrails hardcode a fixed policy list (often 4 to 18 categories) directly into their system prompts. When transferred to a new domain with unfamiliar policies (e.g., academic integrity or corporate HR guidelines), they fail to generalize.
  3. Reasoning Inefficiency and Instability: Recent attempts to train guardrails with reinforcement learning (such as RSafe) often produce verbose reasoning, suffer from repetitive loops, or exhibit cross-lingual language switching.

GSPR bridges this gap by treating safety taxonomies as flexible variables and aligning the guardrail to reason systematically over arbitrary, multi-source safety policies.

Methods

GSPR introduces a flexible training pipeline centered on supervised cold-starting followed by reinforcement learning:

[Flexible Policy Prompt] ---> [Cold-Start SFT] ---> [GRPO RL Alignment] ---> [Accurate, Concise Reasoner]
 (Dynamic Taxonomies)          (1.4k Traces from       (Outcome + Format +
                                Gemini-2.5-Flash)       Category Rewards)
  • Dynamic Taxonomy Prompting: Instead of a fixed instruction template, GSPR’s prompt template accepts a sampled subset of policy definitions as variables. The model is asked to reason within <think> tags before returning structured outputs: \safety{safe/unsafe} and \category{violated policy}.
  • Cold-Start SFT: To jump-start policy comprehension and structured formatting without collapsing into shortcut behavior, the authors distill 1,383 multi-step reasoning traces from Gemini-2.5-Flash across 19 safety taxonomies.
  • Reinforcement Learning via GRPO: Using Group Relative Policy Optimization (eliminating the memory-heavy critic network of PPO), the model is optimized against rule-based rewards:
  • Outcome Rewards ($R^s, R^c$): Binary 0/1 rewards for correct safety and category classifications.
  • Format & Quality Penalty ($R^f$): Enforces correct tag structure, verifies single-language output, uses an $n$-gram checker to penalize repetition, and cuts the reward in half if the model produces verbose, redundant text after the </think> tag.

Datasets

  • Training Data: 19 fine-grained taxonomies covering 167 safety policies. Drawn from six major benchmarks: Aegis, WildGuard, OR-Bench, SafeRLHF, BeaverTails, and GUARDSET-X (specifically its HR and Education subsets).
  • In-Domain Evaluation: Held-out test splits from Aegis, WildGuard, SafeRLHF, and BeaverTails.
  • Out-of-Domain (OOD) Evaluation: Datasets featuring entirely unseen taxonomies: OpenAI Moderation, HEx-PHI, T2T Synthetic Safety Bench, and Do-Not-Answer.

Key Insights and Ideas

  • Prompting matters as much as RL: Simply changing the prompt template to handle dynamic policies ("GSPR safety only") bumped category accuracy from ~25% (RSafe) up to ~72–74%, even before introducing explicit category rewards.
  • Category rewards drastically reduce reasoning cost: Counterintuitively, optimizing for fine-grained category identification alongside binary safety encouraged the model to home in directly on the defining policy violation. GSPR generates much shorter, crisper reasoning traces than binary-only RL guardrails.
  • Cold-start SFT prevents RL degeneration: Starting GRPO directly from a base model (like Qwen3-8B) caused severe language-mixing issues (up to 15.6% English/Chinese switching). A modest cold-start dataset of ~1,400 examples completely stabilized multilingual training.

Key Results

  • In-Domain Moderation: On Qwen3-8B, GSPR reached 86.36% Safety Accuracy (S-Acc) and 77.89% Category Accuracy (C-Acc), outperforming proprietary reasoning models like o3-mini (74.81% S-Acc) and Gemini-2.5-Flash (73.02% S-Acc).
  • Superior Fine-Grained Identification: Baseline guardrails like LLaMA Guard 3 and GuardReasoner scored under 2% C-Acc due to rigid formatting failures, while RSafe scored only 21.71%. GSPR improved category accuracy by over 45% compared to RSafe.
  • Zero-Shot Policy Generalization: On unseen OOD benchmarks, GSPR maintained 93.11% S-Acc and 79.85% C-Acc, demonstrating robust policy grounding rather than memorization.
  • Inference Efficiency: GSPR w/ Cold-start generated explanations in just 34.1 words on Qwen2.5 and 77.7 words on Qwen3, compared to 170–370+ words for baseline reasoning models, reducing latency and compute overhead.

Limitations and Future Work

  • The Category Gap in Complex Scenarios: While binary safety accuracy exceeded 95% on benchmarks like HEx-PHI, category classification lagged at ~58%, indicating that fine-grained boundary distinction remains hard on out-of-distribution distributions.
  • Scope of Coverage: Although trained across 167 policies, many real-world domains (e.g., regional legal statutes, hyper-specific software licenses) remain unrepresented. Future work aims to expand the taxonomy bank to further scale zero-shot policy adaptation.

Summary updated

Abstract

As large language models (LLMs) are increasingly integrated into numerous applications across various domains, LLMs’ safety becomes a critical concern for both application developers and intended users. Currently, great efforts have been made to develop safety benchmarks with fine-grained taxonomies. However, these benchmarks’ taxonomies are disparate with different safety policies. Thus, existing safeguards trained on these benchmarks are either coarse-grained to only distinguish between “safe” and “unsafe,” or constrained by the narrow risk taxonomies of a single benchmark. To leverage these fine-grained safety taxonomies across multiple safety benchmarks, in this paper, we propose GSPR, a Generalizable Safety Policy Reasoner to identify unsafe input prompts and LLMs’ outputs with violated safety taxonomies through Group Relative Policy Optimization (GRPO). Unlike prior safeguards which only cover a fixed set of risk factors, our GSPR incentivizes its reasoning capability with varied safety taxonomies through our careful cold-start strategy and reward design. Consequently, our GSPR can be trained across multiple safety benchmarks with distinct taxonomies and naturally exhibits powerful generalization ability. We conduct extensive experiments to show that our GSPR significantly improves existing safety guardrails’ reasoning capabilities for both safety and category prediction tasks. Moreover, our GSPR not only demonstrates powerful safety generalization abilities but also achieves the least inference token costs with explanations.
BibTeX
@misc{li2025gspr,
  author = {Haoran Li and Yulin Chen and Jingru Zeng and Hao Peng and Huihao JING and Wenbin Hu and Xi Yang and Ziqian Zeng and Sirui Han and Yangqiu Song},
  title = {{GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners}},
  year = {2025},
  eprint = {2509.24418},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24418},
}
Figure