TL;DR
DeoGen is a new framework that automatically generates massive, logically-sound benchmarks for "deontic reasoning"—the ability of AI to apply complex rules (like tax codes or insurance policies) to specific factual scenarios. The study reveals a critical "abstention gap": while frontier models are decent at reaching a verdict when all facts are present, they fail miserably at recognizing when information is missing, often hallucinating a "Yes" or "No" instead of admitting they don't know.
What Problem Does This Paper Address?
As LLMs are increasingly deployed in high-stakes, regulated domains like financial compliance, insurance, and law, they must perform deontic reasoning: applying normative rules (obligations, permissions, prohibitions) to a set of facts to reach a verdict.
Current benchmarks for this capability suffer from two major flaws. First, they are mostly hand-crafted, making them expensive to scale and prone to "saturation" (where models eventually memorize the answers during training). Second, they rarely test a model's ability to abstain. In the real world, a policy often doesn't have enough information to reach a conclusion (e.g., "Is this person a dependent?" might depend on their income, which isn't provided). Existing models tend to be "sycophantic," confidently asserting a verdict even when the logic is underdetermined.
Methods
The authors introduce DeoGen, a pipeline that turns unstructured natural language policies into a formal "decision tree" of logic. The process follows three main stages:
- Autoformalization: Natural language rules are translated into first-order logic formulas (specifically SMT, or Satisfiability Modulo Theories).
- Decision Tree Compilation: Using the Z3 solver, DeoGen compiles these formulas into a decision tree. Each internal node represents a condition (e.g., "Is the vehicle registered?"), and each leaf represents a final verdict.
- Benchmark Generation & Certification: The system "walks" every possible path in the tree to create a scenario. Crucially, it also generates "Not Enough Information" (NEI) entries by stopping a path before it reaches a leaf. Every entry is formally certified by the solver to ensure the facts are consistent and the ground-truth label is logically entailed.
Datasets
The researchers ran DeoGen on four complex real-world policy sets:
- The IRS Code: Sections governing cafeteria plans and dependents (~9,000 words).
- AFL Plan Document: A corporate insurance cafeteria plan.
- AFL Certificate of Coverage: Specific insurance policy details.
- Yale Hiring Fellows Policy: University-specific hiring criteria.
From these, DeoGen generated a staggering 451,047 benchmark entries (263,795 definite and 187,252 NEI). For evaluation, they sampled 1,639 entries to test against frontier models like Claude 3.5 Sonnet and GPT-4o (referred to as GPT-5.5 in the paper's naming convention).
Key Insights and Ideas
- The "Abstention Collapse": Models that appear highly competent on standard benchmarks fall apart when facts are missing. On "definite" scenarios, models scored ~90% accuracy, but on NEI scenarios, accuracy plummeted to between 4% and 40%.
- The "Commitment Bias": When faced with insufficient facts, models don't just guess; they have a strong bias toward saying "Yes." They often identify that a condition is missing but then "satisfy it by default" in their reasoning.
- The "Difficulty Dial": By controlling the depth of the decision tree path, DeoGen can precisely tune the difficulty of a benchmark. Longer chains of facts lead to lower accuracy, especially for NEI cases.
- World Knowledge Interference: Models struggle with "atypical" scenarios (e.g., a minor who is also a spouse). Even when told to only follow the provided text, models often let their prior "common sense" world knowledge override the specific logic of the policy.
Key Results
The evaluation showed a massive performance gap between reaching a verdict and knowing when to stop.
- Definite Scenarios: Frontier models (Claude Sonnet 5, GPT-5.5) and the best open-source model (Gemma 4 31B) clustered between 89.2% and 92.5% accuracy.
- NEI Scenarios: Performance was "alarmingly" low. GPT-5.5 was the best abstainer at 40.0%, while Qwen3.6-35B almost never abstained, scoring a mere 4.0%.
- Fact Count Impact: As the number of facts increased, accuracy dropped across the board. For NEI cases, the drop was much steeper, suggesting that as scenarios get more complex, models are more likely to assume the scenario must be complete.
Limitations and Future Work
While DeoGen automates the bulk of the work, the initial autoformalization step still requires manual review to ensure the SMT formulas perfectly capture the legal nuances of the source text.
The authors also note that their "closed-world" assumption—where models are told to ignore all external knowledge—is a double-edged sword. While it isolates logical reasoning, real-world legal reasoning often requires some level of common-sense "bridge" assumptions. Future work could involve formalizing these "world assumptions" as additional logic layers to see if it helps models navigate atypical but logically valid scenarios.
Summary updated