Abstract
Ensuring the safety and compliance of large language models (LLMs) is of paramount importance. However, existing LLM safety datasets often rely on ad-hoc taxonomies for data generation and suffer from a significant shortage of rule-grounded, real-world cases that are essential for robustly protecting LLMs. In this work, we address this critical gap by constructing a comprehensive safety dataset from a compliance perspective. Using a powerful web-searching agent, we collect a rule-grounded, real-world case dataset OmniCompliance-100K, sourced from multi-domain authoritative references. The dataset spans 74 regulations and policies across a wide range of domains, including security and privacy regulations, content safety and user data privacy policies from leading AI companies and social media platforms, financial security requirements, medical device risk management standards, educational integrity guidelines, and protections of fundamental human rights. In total, our dataset contains 12,985 distinct rules and 106,009 associated real-world compliance cases. Our analysis confirms a strong alignment between the rules and their corresponding cases. We further conduct extensive benchmarking experiments to evaluate the safety and compliance capabilities of advanced LLMs across different model scales. Our experiments reveal several interesting findings that have great potential to offer valuable insights for future LLM safety research.
Introduction: Beyond "Vibe-Based" Safety
Most existing LLM safety datasets suffer from two major flaws: they are synthetic and ad-hoc. Researchers typically create a list of "bad behaviors" (a taxonomy) and then use LLMs to generate examples of those behaviors. While useful for basic alignment, these datasets do not reflect the messy reality of legal requirements, court rulings, or corporate policies.
As LLMs are integrated into high-stakes sectors like finance, medicine, and law, safety is no longer just about avoiding offensive language; it is about compliance. There is a critical shortage of data that connects specific, authoritative rules (e.g., Article 5 of the EU AI Act) to documented real-world incidents. Researchers have introduced OmniCompliance-100K to bridge this gap, providing the first large-scale dataset of over 106,000 real-world safety cases grounded in nearly 13,000 specific regulatory rules.
Methodology: The Agentic Pipeline
The authors developed a two-stage "agentic" pipeline to bridge the gap between abstract rules and real-world evidence:
- Rule Curation: A team curated 74 regulations and policies across 9 domains, structuring them into hierarchical trees. This resulted in 12,985 distinct rules.
- Agentic Case Searching: Instead of manual searching, they built a web-searching agent. The agent analyzes a rule, generates multiple search queries, crawls authoritative sources (official reports, court records, news), and then filters and summarizes the findings into a standardized format.
- Knowledge Graph Construction: They linked these rules and cases into a Rule-Case Knowledge Graph, allowing for multi-hop reasoning to see how a single real-world incident might violate multiple rules across different jurisdictions.
The OmniCompliance-100K Dataset
The primary contribution, OmniCompliance-100K, represents a significant leap in scale and authenticity compared to predecessors:
- Scale: 12,985 rules and 106,009 associated real-world cases.
- Domains: Covers AI Safety Law (EU AI Act, SB 53), Data Privacy (GDPR, HIPAA, CCPA), Chinese Law (PIPL), Platform Policies (OpenAI, Google, X, Reddit), Education, Finance, Medical Law, and Human Rights.
- Authenticity: Unlike predecessors like Air-Bench or GuardSet-X, which rely on synthetic cases, OmniCompliance-100K uses 100% real-world documented events.
Benchmarking and Key Results
The authors benchmarked a wide array of models (Grok, GPT-4o-Mini, DeepSeek-V3.2, Qwen, Llama) using a 2-way classification task (Permitted vs. Prohibited).
Key Findings
- Top Performers: GLM-4.5 and DeepSeek-V3.2 led the pack, with average macro-F1 scores around 92-93%.
- The "Guardrail" Paradox: Models specifically trained to be safety filters (like Llama-Guard-3 or WildGuard) performed surprisingly poorly. This suggests these models are overfitted to narrow, synthetic safety categories and fail to generalize to actual regulatory compliance.
- The 3B Sweet Spot: A performance threshold was identified at the 3-billion parameter mark. Models like Qwen2.5-3B-Instruct punch significantly above their weight, while models below 1.5B show a precipitous drop in compliance reasoning.
- Domain Specificity: Models showed exceptionally high reliability (95-97% F1) in EU financial regulations, but struggled with the EU AI Act's "Prohibited AI Practices" (Chapter II), where even the best models often stayed below 80%.
- The Difficulty of Discrimination: The "Education: Bias and Discrimination" category was the hardest for every model tested, highlighting that identifying subtle societal biases in real-world cases remains a major hurdle for frontier models.
Limitations and Future Work
While OmniCompliance-100K provides a rigorous new benchmark, it has limitations. Human validation was performed on a subset of 2,220 cases due to the high cost of legal expertise. Additionally, while the authors used filtering and anonymization, sourcing data from the open web carries a residual risk of containing personally identifiable information (PII). Future work will move beyond classification to "compliance reasoning," requiring models to explain why a case violates a specific sub-clause of a law using the provided knowledge graph.
Summary updated