Legal codes, clinical guidelines and tax rules are built from defaults and exceptions. A general rule says a taxpayer may take a deduction. A more specific provision takes it away. A third provision sometimes restores it for a different reason. When a case falls under two provisions that point in opposite directions and neither is more specific, the correct answer is "the rules do not settle this." Forty years ago, knowledge-representation researchers worked out exactly how such conflicts should be resolved when a concept inherits from several parents that disagree. They gave the failure modes names: specific rules overriding general ones, symmetric standoffs, defeated rules that still block others ("zombie paths"), and reviving a defeated rule for the wrong reason ("reinstatement"). Today's language models fail at statutory reasoning, and the documented failures look like these same patterns. Yet no benchmark tests those patterns directly.
We propose a contamination-free generator of statute-like rule systems. Each item is built from a known conflict pattern taken from nonmonotonic inheritance theory, meaning the logic of defaults that newer, more specific information can override. Ground truth comes from a polynomial-time skeptical inheritance algorithm. We use the generator to diagnose which resolution strategy a model actually follows: correct specificity, "nearest rule wins," credulous guessing, or reinstatement. Before any failure is labeled a reasoning failure, the evaluation protocol rules out answer-calibration artifacts. The central test is whether these diagnoses predict per-item errors on a real statutory benchmark, SARA. We also test whether handing the model's extracted rule graph to a skeptical solver closes the gap, and whether that gain depends on the solver's semantics rather than on the act of translating to code.
Core Claim
Existing synthetic statute tests vary only width and depth of tree-shaped definitions. That design cannot distinguish a model that resolves exceptions by genuine specificity from one that uses surface proximity: the rule mentioned closest to the conclusion, or the one with the fewest cross-references between it and the case. We claim three things. First, the difference shows up sharply on multi-parent conflict structures. Second, it shows up in invariance tests: adding an irrelevant intermediate definition, or a redundant cross-reference, should never change the answer. Third, a model's error profile on these structures explains a meaningful share of its errors on real statutes. If the third claim fails, the benchmark is a curiosity. If it holds, it gives the field a reproducible, structural account of why models over-apply statutes.
Grounding in Prior Work
Three statutory-reasoning papers motivate the gap. A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering built SARA from nine Internal Revenue Code sections. It showed that a hand-written Prolog translation solves it perfectly, while BERT and Legal BERT sit near chance on entailment. Its ablations suggest these models lean on case-fact cues rather than on the statutes. Factoring Statutory Reasoning as Language Understanding Challenges decomposed the task into argument identification, argument coreference, structure extraction and argument instantiation. Its pipeline, however, caps dependency depth and cannot backtrack after a misread exception. Can GPT-3 Perform Statutory Reasoning? is the closest existing work. It introduced synthetic statutes to escape memorization and reported that GPT-3 over-concludes that statutes apply. Its synthetic rules are exception-light trees parameterized by width and depth, with no diamonds, preemption chains or zombie paths. We reuse its core move, invented rules absent from pretraining data, and replace its tree-shaped structural axis with a structural axis based on conflict patterns.
The method's semantics come from the inheritance literature. Implicit Ordering of Defaults in Inheritance Systems defined inferential distance: class A beats class B when there is a path showing A is a subclass of B. The same paper proved that shortest-path resolution breaks under inserted intermediate nodes and redundant links. We turn both of those breakages into invariance tests. A Skeptical Theory of Inheritance in Nonmonotonic Semantic Networks gives a unique, polynomial-time skeptical conclusion set for acyclic networks, built bottom-up from the instance with preemption and neutralization. It serves as our ground-truth oracle. A Skeptic's Menagerie catalogs the compound interactions that fast marker-passing gets wrong, including zombies and the reinstatement trap. It supplies the phenotypes, meaning the named conflict patterns, that our generator plants.
Two constraints shape the design. The skeptical theory is defined only for acyclic graphs, so the generator stays acyclic and we do not claim coverage of circular cross-references. The domain literature also warns that supplying rules is not the same as using them. TheoremQA found theorem augmentation gave no significant boost, and LegalBench documents models reaching conclusions without a coherent rule application. This is why we measure reasons through paired counterfactual items, not only final answers.
The reproducibility lens comes from Automated Medical Coding on MIMIC-III and MIMIC-IV. That audit found that default 0.5 thresholds and a miscomputed macro F1 manufactured apparent architectural differences, and that correcting the metric doubled reported scores. We apply the same discipline to reasoning claims. A model that says "applies" too often may be miscalibrated rather than logically broken, so every headline number is reported after per-model calibration of the decision and abstention thresholds.
Research Program
The generator samples acyclic inheritance networks over invented legal categories, such as "a qualifying brennard" or "a covered ostral transfer." It renders them as statute-like prose with numbered subsections, cross-references and "except as provided in" clauses. Each item plants exactly one target phenotype. The first is a simple preemption, where a specific exception overrides a general rule. The second is a symmetric diamond, where the case falls under two unordered provisions with opposite effects, so the correct answer is "undetermined." The third is an on-path preemptor versus off-path conflictor, where one rule is defeated by specificity and another conflict is merely balanced. The fourth is a zombie path, where a defeated provision still neutralizes a conclusion higher up. The fifth is a reinstatement trap, where a general rule is defeated but a separate specific rule yields the same conclusion.
For every item the generator also emits paired variants. Some are invariance variants: an irrelevant intermediate definition inserted, or a redundant cross-reference added. Others are counterfactual variants, such as removing the separate specific rule in a reinstatement item. A model that is right for the right reason must flip its answer when that rule is removed. We label each item with the skeptical answer from the Horty–Thomason–Touretzky algorithm, and separately with the set of credulous extensions. Credulous-but-consistent answers are therefore distinguishable from errors. Answers are three-way: applies, does not apply, or undetermined. Depth, width, number of distractor rules and surface ordering of provisions are controlled, and the generator, seeds and rendering templates are released.
The first analysis fits each model's answers to a small set of candidate resolution strategies and reports which fits best, per phenotype. The candidates are skeptical inheritance, credulous choice, shortest-path or textual-proximity resolution, and marker passing with reinstatement. The second analysis is the transfer test. Two annotators independently tag each SARA case by whether its gold answer depends on an exception preempting a general rule, and by the structural pattern involved. We then test whether a model's synthetic error rate on the matching phenotype predicts its SARA error on those cases. The comparison is against a model that uses only width and depth features taken from the Blair-Stanek et al. design.
The third analysis is an intervention. The model extracts the rule graph from the text in the spirit of the Holzenberger and Van Durme decomposition, but without their depth cap. A solver then computes the answer, and we vary the solver's semantics while holding the extraction fixed.
Baselines
The naive reference points are "always applies," which matters because false positives are the documented failure mode, and majority class per phenotype. Any model near these on diamonds or zombies is not handling conflict at all.
The strongest existing method on synthetic statutes is few-shot prompting as in Can GPT-3 Perform Statutory Reasoning?. That paper reports 78% zero-shot accuracy and 100% two-shot accuracy on 2-wide, 2-deep statutes, and 81% two-shot on 3-wide, 3-deep statutes. On SARA itself the strongest reported learned result is GPT-3 at up to 71% accuracy, against 100% for the hand-built Prolog solver from A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering. We found no reported result on Lacuna for any model on conflict-pattern synthetic statutes, so our phenotype-level numbers will be the first.
We run chain-of-thought and two-shot prompting on open-weight models named in AppealCase, LLaMA-3.3 and DeepSeek-R1, plus one closed model. Open weights keep the evaluation reproducible. All prompting conditions receive the same exemplar pool, the same number of shots and the same answer-parsing code. Every condition, including ours, gets its thresholds calibrated on a shared held-out development split.
The key ablation holds the model's extracted graph fixed and swaps the solver. The options are skeptical inheritance, Prolog-style negation-as-failure with clause ordering, shortest-path resolution, and marker passing that allows reinstatement. Beating the few-shot baseline means two things. On width/depth-matched items, the skeptical-solver pipeline must exceed it on diamonds, zombies and reinstatement counterfactuals by more than the seed-to-seed spread. On SARA, it must also exceed 71%. If the Prolog solver matches the skeptical solver, the gain comes from translation and not from inheritance semantics.
Why not just…?
Why not just use a stronger reasoning model with chain-of-thought? Blair-Stanek et al. found that step-by-step prompting gave only marginal gains and that accuracy fell as statutes grew. LegalBench shows conclusions outrunning application. Neither result rules out newer reasoning models, so we include DeepSeek-R1. The paired counterfactual items test the answer honestly. If a reasoning model passes the invariance and reinstatement counterfactuals at skeptical-oracle rates, the intervention half of this proposal is unnecessary. The benchmark would remain useful as a check that such models reason by specificity.
Why not just have the model write Prolog, as SARA's solver did? This is the real threat. Prolog with ordered clauses and negation-as-failure often yields the specific-beats-general answer. It has no native notion of a balanced standoff, though, so on diamonds it commits to whichever clause comes first. It can also reinstate rules. The solver-swap ablation settles the question directly. If Prolog matches skeptical inheritance even on diamond and zombie items, we drop the semantic claim and report that translation alone is sufficient.
Why not just calibrate the model's "applies" threshold? The medical-coding audit shows that calibration alone can erase apparent differences, so the concern is serious. We calibrate every condition before comparing. If the false-positive gap on preemption items disappears after calibration, the finding is that models are miscalibrated rather than nonmonotonically blind. That is a publishable negative result for the "reasoning deficit" story. Calibration cannot, however, repair invariance violations, because the underlying rules are identical across paired items, so those remain a clean test.
Why not just extend width and depth in the existing synthetic statutes? Tree-shaped statutes with single exceptions have no diamonds, so "undetermined" never arises and proximity heuristics are never punished. The transfer test compares phenotype features against width/depth features as predictors of SARA errors. If width and depth predict equally well, the phenotypes add nothing, and we will say so.
Decisive Checks
The proposal succeeds only if three results hold. First, some model family shows a phenotype-specific signature, such as correct simple preemption but proximity-driven answers under inserted intermediate definitions, that survives threshold calibration. Second, phenotype error rates predict SARA per-item errors beyond width and depth. Third, the skeptical solver beats the Prolog solver on diamonds and zombies given identical extracted graphs.
As a secondary calibration check, we apply per-model threshold tuning to AppealCase first-instance reversal prediction. There all 20 reported models score below 50% F1. The check asks whether that wall moves once calibration is controlled, which tells us how much of the reported "reasoning failure" in legal benchmarks is a decision-boundary artifact.
References
- A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
- Factoring Statutory Reasoning as Language Understanding Challenges
- Can GPT-3 Perform Statutory Reasoning?
- Implicit Ordering of Defaults in Inheritance Systems
- A Skeptical Theory of Inheritance in Nonmonotonic Semantic Networks
- A Skeptic's Menagerie: Conflictors, Preemptors, Reinstaters, and Zombies in Nonrnonotonic Inheritance
- TheoremQA: A Theorem-driven Question Answering Dataset
- LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios