Deogen: Automatic Benchmark Generation for Reasoning over Policies

Sam Kouteili, William Liang, Keeran Murray, Adriana Zhou, Ramesh Viswanathan, Scott Shapiro, Ruzica PiskacarXiv 2026 · arXiv

TL;DR

DeoGen is a new framework that automatically generates massive, logically-sound benchmarks for "deontic reasoning"—the ability of AI to apply complex rules (like tax codes or insurance policies) to specific factual scenarios. The study reveals a critical "abstention gap": while frontier models are decent at reaching a verdict when all facts are present, they fail miserably at recognizing when information is missing, often hallucinating a "Yes" or "No" instead of admitting they don't know.

What Problem Does This Paper Address?

As LLMs are increasingly deployed in high-stakes, regulated domains like financial compliance, insurance, and law, they must perform deontic reasoning: applying normative rules (obligations, permissions, prohibitions) to a set of facts to reach a verdict.

Current benchmarks for this capability suffer from two major flaws. First, they are mostly hand-crafted, making them expensive to scale and prone to "saturation" (where models eventually memorize the answers during training). Second, they rarely test a model's ability to abstain. In the real world, a policy often doesn't have enough information to reach a conclusion (e.g., "Is this person a dependent?" might depend on their income, which isn't provided). Existing models tend to be "sycophantic," confidently asserting a verdict even when the logic is underdetermined.

Methods

The authors introduce DeoGen, a pipeline that turns unstructured natural language policies into a formal "decision tree" of logic. The process follows three main stages:

  1. Autoformalization: Natural language rules are translated into first-order logic formulas (specifically SMT, or Satisfiability Modulo Theories).
  2. Decision Tree Compilation: Using the Z3 solver, DeoGen compiles these formulas into a decision tree. Each internal node represents a condition (e.g., "Is the vehicle registered?"), and each leaf represents a final verdict.
  3. Benchmark Generation & Certification: The system "walks" every possible path in the tree to create a scenario. Crucially, it also generates "Not Enough Information" (NEI) entries by stopping a path before it reaches a leaf. Every entry is formally certified by the solver to ensure the facts are consistent and the ground-truth label is logically entailed.
Figure 1: DeoGen pipeline, illustrating transformation from text to a decision structure and its tabular representation.

Datasets

The researchers ran DeoGen on four complex real-world policy sets:

  • The IRS Code: Sections governing cafeteria plans and dependents (~9,000 words).
  • AFL Plan Document: A corporate insurance cafeteria plan.
  • AFL Certificate of Coverage: Specific insurance policy details.
  • Yale Hiring Fellows Policy: University-specific hiring criteria.

From these, DeoGen generated a staggering 451,047 benchmark entries (263,795 definite and 187,252 NEI). For evaluation, they sampled 1,639 entries to test against frontier models like Claude 3.5 Sonnet and GPT-4o (referred to as GPT-5.5 in the paper's naming convention).

Key Insights and Ideas

  • The "Abstention Collapse": Models that appear highly competent on standard benchmarks fall apart when facts are missing. On "definite" scenarios, models scored ~90% accuracy, but on NEI scenarios, accuracy plummeted to between 4% and 40%.
  • The "Commitment Bias": When faced with insufficient facts, models don't just guess; they have a strong bias toward saying "Yes." They often identify that a condition is missing but then "satisfy it by default" in their reasoning.
  • The "Difficulty Dial": By controlling the depth of the decision tree path, DeoGen can precisely tune the difficulty of a benchmark. Longer chains of facts lead to lower accuracy, especially for NEI cases.
  • World Knowledge Interference: Models struggle with "atypical" scenarios (e.g., a minor who is also a spouse). Even when told to only follow the provided text, models often let their prior "common sense" world knowledge override the specific logic of the policy.

Key Results

The evaluation showed a massive performance gap between reaching a verdict and knowing when to stop.

  • Definite Scenarios: Frontier models (Claude Sonnet 5, GPT-5.5) and the best open-source model (Gemma 4 31B) clustered between 89.2% and 92.5% accuracy.
  • NEI Scenarios: Performance was "alarmingly" low. GPT-5.5 was the best abstainer at 40.0%, while Qwen3.6-35B almost never abstained, scoring a mere 4.0%.
  • Fact Count Impact: As the number of facts increased, accuracy dropped across the board. For NEI cases, the drop was much steeper, suggesting that as scenarios get more complex, models are more likely to assume the scenario must be complete.
Figure 3: Accuracy as a function of the number of facts shown. Path depth acts as a difficulty dial.

Limitations and Future Work

While DeoGen automates the bulk of the work, the initial autoformalization step still requires manual review to ensure the SMT formulas perfectly capture the legal nuances of the source text.

The authors also note that their "closed-world" assumption—where models are told to ignore all external knowledge—is a double-edged sword. While it isolates logical reasoning, real-world legal reasoning often requires some level of common-sense "bridge" assumptions. Future work could involve formalizing these "world assumptions" as additional logic layers to see if it helps models navigate atypical but logically valid scenarios.

Summary updated

Abstract

Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.
BibTeX
@misc{kouteili2026deogen,
  author = {Sam Kouteili and William Liang and Keeran Murray and Adriana Zhou and Ramesh Viswanathan and Scott Shapiro and Ruzica Piskac},
  title = {{Deogen: Automatic Benchmark Generation for Reasoning over Policies}},
  year = {2026},
  eprint = {2610.38687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38687},
}
Figure