When a statute says "employee," it rarely means what the word means in everyday English, and it may not mean what the same word means two chapters later. Legislatures handle this with definition clauses ("As used in this chapter, 'employee' means…"). Each clause binds a surface word to a specific meaning within a specific part of the code. Language models reading retrieved law tend to ignore these bindings. They fall back on the general-language sense of the word, or on whichever definition happens to be nearby. We propose to treat every legally defined term in a statute the way a compiler treats an identifier in a program. When a model translates a provision into logic rules, each defined term it uses must be resolved to exactly one in-scope definition section. A cheap exact-lookup checker verifies that resolution against an index of definitions extracted from the code itself.
In two sentences: retrieval for legal reasoning should fetch more than passages similar to the question. It should also fetch the definitions that govern each technical word in those passages, chosen by the code's own scoping rules. Writing the provision as checkable logic, with every predicate tied to a definition section, makes that requirement enforceable and auditable rather than hoped for. We test whether this binding step is what lifts legal question answering past the plateau that retrieval alone reaches, and whether the formal logic matters or the binding alone does the work.
Why this gap, and why now
Three findings motivate the proposal. AI for Statutory Simplification builds a large corpus of U.S. state statutes (StateCodes) and 3,700 expert yes/no questions on state unemployment insurance law (LaborBench). It finds that retrieval raises F1 by about 0.18 over models without retrieval, yet peaks at 0.691, with the answer wrong roughly a third of the time even when the law is in context. F1 also swings from 0.52 in Virginia to 0.81 in Arkansas. Unemployment insurance law is exactly where terms such as "base period," "wages," and "employee" are defined differently state by state. That makes it a natural home for this failure.
DOLFIN shows the same pattern in another domain. Strong models translate financial "charges" as criminal charges, a case of the dominant pre-training sense overriding the domain sense. Smaller models get worse, not better, when given more surrounding text. Agent-Specific Deontic Modality Detection shows that even "shall" changes legal force depending on which party it attaches to. All three point to one problem: the meaning of a legal word is fixed by something outside the sentence, and current pipelines do not fetch that something explicitly.
The method comes from mathematics. Direct Dependency Retrieval translates informal math into Lean, a proof language. The model names the library definitions it needs directly, and an exact suffix-array lookup checks that those names exist. Recall of the right dependencies rises from 0.02–0.10 with embedding search to 0.80–0.90, and invented names fall from about 30% to under 2%. Legal definitions are harder in one respect that matters here. In Lean a name is globally unique. In a statute the same string is bound to different definitions depending on where in the code you stand. Domain polysemy in law is therefore a scope-resolution problem. Solving that scope-resolution problem is the new piece.
Core claim
When a model translates a statutory provision into logic rules, forcing every legally defined predicate to carry a pointer to its governing definition section reduces errors caused by default-sense interpretation. The pointer is resolved by the code's own scoping language and checked by exact lookup. Such errors account for a measurable share of the residual LaborBench errors. We further claim that formalization is what makes the binding enforceable. A model can be told to "consult the definitions" in free text, but only a program with named predicates can be mechanically checked for unbound or mis-scoped terms.
Research program
Definition index. We first build a definition index over StateCodes and the Internal Revenue Code sections used in SARA. Definition clauses follow a small set of drafting patterns ("means," "includes," "does not include," "as used in this section/chapter/title"). We extract each clause as a triple of term, scope node, and section identifier. The code is represented as a hierarchy of title, chapter, and section, in the spirit of the document tree that ParCleanse builds from network protocol specifications. Scope resolution is a deterministic rule: the most specific enclosing definition wins, and explicit cross-references override that default. Extraction quality is measured on a hand-checked sample before anything downstream is trusted.
Translation step. Next comes the translation step. Given retrieved operative provisions, a language model writes Horn clauses, which are if-then logic rules of the kind used in Prolog. It must annotate each predicate either with a definition section identifier or with the explicit label "undefined/ordinary meaning." A checker then does three things. It verifies that the identifier exists, which is the analogue of the suffix-array check. It verifies that the identifier's scope covers the operative provision. It verifies that no more-specific in-scope definition was skipped. Failures return to the model with the correct candidate set, and bound definitions are themselves translated recursively up to a fixed depth. Predicates labeled "ordinary meaning" are where open-textured judgment lives. They are evaluated by the model and logged, so a user can see exactly which conclusions rest on unverified interpretation.
Answering. The resulting program is run in Prolog, or in Z3 (a constraint solver) when arithmetic is involved. For SARA, the fact pattern is translated and the query is executed. For LaborBench, whose questions ask what a state's law provides, the question is translated into a query over the program. A parallel non-formal path feeds the same bound definitions to the generator as text. This path is the key ablation.
This differs from the closest systems along clear lines. L4L translates criminal statutes into Horn clauses checked by Z3. It works from a fixed, pre-formalized set of articles with no retrieval and no definition scoping. Compliance Checking for Public Administration Processes sketches retrieval feeding temporal logic and solvers, but reports no implementation or results. Factoring Statutory Reasoning decomposes statutory reasoning into argument identification, coreference, structure extraction, and instantiation. It treats coreference within a text, whereas we treat binding of terms to definitions elsewhere in the code.
Baselines
The naive reference point is the same generator model answering without retrieval. Its relevant number is that retrieval adds an average of 0.18 F1 on LaborBench, according to AI for Statutory Simplification.
The strongest existing method is that paper's best retrieval pipeline, which reports 0.691 F1 on LaborBench (Gemini retriever with DeepSeek V3 generator). It is the strongest competitor because it is tuned on exactly our target benchmark. We will rerun it with the same generator as our system, the same retrieved operative passages, and the same token budget per question. Beating it means a statistically reliable gain over the rerun number on the full benchmark, with a larger gain on the questions our annotators mark as definition-dependent.
On the retrieval side, we use LeSeR, which re-ranks dense-retrieval candidates with BM25. It reports mAP@10 of 0.6655 on ObliQA against 0.1077 for dense retrieval alone. Because it is the best evidence that exact lexical matching fixes legal retrieval, we will also apply it directly to the definition index as a competitor binder. Under that competitor, the top BM25-reranked definition clause for each term is chosen with no scope rule.
The key ablation separates three ingredients. The first variant keeps the binding but drops the formalization: bound definitions are passed as text to the generator. The second keeps the formalization but drops the binding, which is L4L-style Horn clauses with free predicates. The third keeps binding and formalization but drops the scope check, choosing definitions by retrieval rank alone. All variants share the generator, the retrieved operative text, and a matched number of model calls, and each gets the same small prompt-tuning budget on a held-out development split of LaborBench states.
Why not just…?
Why not just put the whole chapter, definitions included, in a long context window? This is a real threat on LaborBench, where a state's unemployment chapter often fits in context. Two findings argue against it. The 0.691 ceiling already holds with the relevant law in context (AI for Statutory Simplification), and smaller models degrade when given section-level context (DOLFIN). We will run full-chapter prompting as a baseline, and on the synthetic benchmark we will vary the distance between a term's use and its governing definition. If long context matches binding at all distances, the scope checker is unnecessary and we will say so.
Why not just retrieve definitions with hybrid lexical search and skip the logic? This is the most plausible competitor, and the first ablation above tests it directly. LeSeR's results suggest exact-match retrieval will find candidate definitions. Rank alone cannot choose between two clauses that define the identical string at different scopes, so we expect it to fail specifically on multi-definition terms. If text-only binding with the scope rule matches the formal pipeline, the honest conclusion is that scoped binding, not formalization, is the contribution. The formal variant would then be justified only by its auditability and abstention signal.
Why not just prompt the model to reason step by step and look up definitions itself? Can GPT-3 Perform Statutory Reasoning? found step-by-step prompting gave marginal gains and left a persistent bias toward declaring that a statute applies. We will include a tool-using variant that can query the definition index freely without being forced to bind. This separates "having access to definitions" from "being required to use the right one."
Proposed experiments
Synthetic code test. The decisive controlled test extends the synthetic-statute method of Can GPT-3 Perform Statutory Reasoning?. That paper reports 81% two-shot accuracy on 3-wide, 3-deep invented statutes. We generate invented codes in which a term is defined two or three times at different scopes, and some definitions deliberately contradict the term's everyday sense. Memorization cannot help here, and the ground truth comes from Prolog. We predict that baselines lose accuracy roughly in proportion to the share of uses whose governing definition differs from the ordinary sense, while the bound pipeline stays flat. We also predict a falling false-positive rate. A flat curve for long-context prompting would falsify the need for binding.
Real-law tests. On real law we evaluate LaborBench F1 overall and on a subset of several hundred questions annotated by law students with their governing definition sections. That subset lets us also score binding accuracy directly, for which no reported result was found on Lacuna. We also evaluate SARA accuracy, where the reported reference points are 100% for the hand-written Prolog program and 71% for GPT-3 (A Dataset for Statutory Reasoning in Tax Law; Can GPT-3 Perform Statutory Reasoning?). We additionally check whether the state-to-state F1 spread shrinks, since our account predicts that variance tracks how heavily each state relies on chapter-specific definitions.
Diagnostic and secondary test. As a diagnostic, we check whether the fraction of predicates labeled "ordinary meaning" predicts errors well enough to serve as an abstention signal. As a secondary test of whether the idea transfers to agent-conditioned terms, we bind "shall" clauses to the party they obligate and score the result against LEXDEMOD labels. The reference point there is RoBERTa-Large at 76.71 F1 without known triggers.
References
- AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark
- DOLFIN - Document-Level Financial Test-Set for Machine Translation
- Agent-Specific Deontic Modality Detection in Legal Language
- Improving Autoformalization Using Direct Dependency Retrieval
- Validating Network Protocol Parsers with Traceable RFC Document Interpretation
- Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning
- Compliance Checking for Public Administration Processes using Retrieval-Augmented Generation in LLMs: Novel Directions and Challenges
- Factoring Statutory Reasoning as Language Understanding Challenges
- 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering
- Can GPT-3 Perform Statutory Reasoning?
- A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering