Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance

Wenbin Hu, Huihao Jing, Haochen Shi, Haoran Li, Yangqiu SongSubmission to ICLR 2026 · Withdrawn

TL;DR

Current LLM safety guardrails rely on ad-hoc taxonomies that fail to address complex, real-world deployment risks with legal rigor. This paper introduces safety compliance, reframing AI safety around comprehensive legal frameworks—specifically the EU AI Act and GDPR—to build a systematic benchmark and train Compliance Reasoner, an 8B model fine-tuned with Group Relative Policy Optimization (GRPO) that significantly outperforms standard models and existing safety guardrails.

What Problem Does This Paper Address?

Most LLM safety solutions depend on informal, handcrafted harm taxonomies (such as generic categories for "hate speech" or "misinformation") or narrow courtroom legal benchmarks (like specialized data-transfer disputes). These ad-hoc boundaries struggle to systematically regulate autonomous agents in high-stakes fields like finance, healthcare, and law.

The authors propose grounding LLM safety directly in established statutory regulations. By treating comprehensive legal frameworks—namely the EU Artificial Intelligence Act (EU AI Act) and the General Data Protection Regulation (GDPR)—as standardized safety taxonomies, practitioners gain an exhaustive, legally anchored rubric to define and verify safe AI behaviors.

Methods

The authors introduce an end-to-end pipeline covering benchmark generation, reasoner training, and data extrapolation:

  1. Statute-to-Scenario Benchmark Synthesis: Legal frameworks are modeled as hierarchical trees ($T = (V, E)$), where each node is a discrete regulatory clause. Root-to-leaf paths are extracted to form complete regulatory chains, which serve as seed prompts for DeepSeek-V3.1 to generate balanced "permitted" vs. "prohibited" real-world LLM scenarios complete with factual backgrounds, legal issues, arguments, and jurisdictions.
  2. Compliance Reasoner Training:
  3. Cold-Start SFT: A base model (Qwen3-8B) is fine-tuned on step-by-step reasoning trajectories distilled from DeepSeek-V3.1, enforcing a structured output: <think> [reasoning trace] </think> \boxed{verdict}.
  4. Reinforcement Learning via GRPO: The cold-started model is further trained using Group Relative Policy Optimization (GRPO) with a rule-based reward function combining binary compliance correctness ($R_{\text{comply}}$) and format adherence ($R_{\text{format}}$): $$R(o|q) = R_{\text{format}}(o) \cdot (R_{\text{comply}}(o|q) + \alpha)$$
  5. Safety Data Extrapolation: The trained reasoner classifies existing generic safety benchmarks into corresponding regulatory chapters and generates detailed, legally grounded compliance scenarios from ordinary safety seeds.

Datasets

  • Safety Compliance Benchmark (Novel): Synthesized directly from legal statutes, yielding 1,684 balanced scenarios for the EU AI Act and 1,012 scenarios for the GDPR (split 50:50 into train/test sets).
  • Existing Safety Datasets (for Extrapolation): Sourced from Aegis-2.0, WildGuard, SafeRLHF, and OpenAI Mod test sets to test cross-domain generalization into compliance frameworks.

Key Insights and Ideas

  • Standard guardrails fail at compliance: Widely used guardrails like Llama-Guard-3-8B perform around 48% accuracy on legal compliance tasks—essentially random guessing on binary decisions—demonstrating that common safety tuning does not transfer to statutory requirements.
  • Hierarchical tree traversal prevents fragmented rules: Extracting entire root-to-leaf regulatory paths ensures scenarios reflect contextual nuances (such as specific law-enforcement exemptions under Article 5(1)(h) of the EU AI Act) rather than isolated clauses.
  • Pre-existing safety datasets map naturally to specific statutes: When mapped onto legal chapters, generic safety benchmarks disproportionately align with prohibited practices (EU AI Act Chapter II, GDPR Chapter II) and penalty/processing provisions (EU AI Act Chapter XII/XIII, GDPR Chapter IX), confirming that traditional safety is an informal subset of broader regulatory law.

Key Results

  • EU AI Act Benchmark: Compliance-Reasoner-GRPO achieved 66.86% accuracy, outperforming the Qwen3-8B base model (56.41%, a +10.45% gain), general-purpose baselines (GPT-4o-mini at 57.01%, DeepSeek-V3.1 at 59.03%), and guardrails (Llama-Guard-3-8B at 48.34%, Guard-Reasoner at 53.21%).
  • GDPR Benchmark: Compliance-Reasoner-GRPO reached 77.27% accuracy, achieving an +11.85% improvement over base Qwen3-8B (65.42%) and outperforming all baselines (Llama-Guard-3-8B scored 48.22%).
  • Generalization across backbones: Applying the same pipeline to Qwen2.5-7B-Instruct yielded consistent gains of +7.12% on the EU AI Act and +11.64% on GDPR.
  • Data Quality: Human evaluation by domain experts scored synthesized scenarios above 95% across legal alignment, scenario coherence, and relevance to LLM safety contexts.

Limitations and Future Work

  • Jurisdictional Scope: The current work exclusively focuses on European legislation (EU AI Act and GDPR), leaving out regulatory standards from other regions (e.g., US executive orders, HIPAA, or Asian AI governance frameworks).
  • Binary Simplification: Compliance evaluations are structured as binary ("permitted" vs. "prohibited") classification, whereas real-world legal assessments often deal with jurisdictional ambiguities, unsettled case law, and probabilistic liability risks.
  • Synthetic Data Reliance: The primary dataset generation depends on frontier LLM outputs (DeepSeek-V3.1), which, despite human spot-checking, may inherit model-specific priors or biases.

Summary updated

Abstract

The proliferation of Large Language Models (LLMs) has demonstrated remarkable capabilities, elevating the critical importance of LLM safety. However, existing safety methods rely on ad-hoc taxonomy and lack a rigorous, systematic protection, failing to ensure safety for the nuanced and complex behaviors of modern LLM systems. To address this problem, we solve LLM safety from legal compliance perspectives, named **safety compliance**. In this work, we posit relevant established legal frameworks as safety standards for defining and measuring safety compliance, including the EU AI Act and GDPR, which serve as core legal frameworks for AI safety and data security in Europe. To bridge the gap between LLM safety and legal compliance, we first develop a new benchmark for safety compliance by generating realistic LLM safety scenarios seeded with legal statutes. Subsequently, we align Qwen3-8B using Group Policy Optimization (GRPO) to construct a safety reasoner, **Compliance Reasoner**, which effectively aligns LLMs with legal standards to mitigate safety risks. Our comprehensive experiments demonstrate that the Compliance Reasoner achieves superior performance on the new benchmark, with average improvements of +10.45% for the EU AI Act and +11.85% for GDPR.
BibTeX
@misc{hu2025safety,
  author = {Wenbin Hu and Huihao JING and Haochen Shi and Haoran Li and Yangqiu Song},
  title = {{Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance}},
  year = {2025},
  eprint = {2509.22250},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22250},
}
Figure