Training Small Statutory Parsers That Cannot Launder Their Answers

A promising way to make automated legal decisions auditable is to split the work in two. A language model reads a case and writes down the facts as a small logic program. A deterministic rule engine (Prolog, a logic-programming language) then applies the statute to those facts. The answer is computed by the engine, so the audit trail is exact. That trail, however, is only as honest as the facts the model wrote down. If we train a compact model with reinforcement learning, rewarding it whenever the engine's answer is right, the model can learn to write fact programs that reach the right verdict through wrong facts. It might invent a dependent, drop an income item, or misread a date. Those programs would pass a correctness check and still mislead an auditor. This is the familiar "fairwashing" problem of persuasive but unfaithful explanations, moved from post-hoc surrogates into the parser itself.

We propose to train 3B–7B models as statutory fact parsers using a solver-executed reward with one added ingredient: an edit-equivariance reward. We minimally edit the case text, for example changing a salary figure, a birth year, or a marital status, and require the model's fact program to change in exactly the corresponding fact and nowhere else. The engine then recomputes the answer. This check needs no gold annotations, since the edit generator knows what it changed. It also sidesteps the hardness of counterfactual explanation: we do not search for counterfactuals, we create them on the input side and execute. The core claim is that answer-only execution reward raises accuracy but also raises the rate of laundered-correct answers. Adding edit-equivariance should remove most of that laundering at little or no cost in accuracy, which yields a small, cheap model whose audit trails can be contested fact by fact.

Where This Sits

Three lines of prior work motivate the proposal. The SARA tax-law dataset (A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering) applies nine U.S. Internal Revenue Code sections to 376 hand-written cases. It shows that a hand-written Prolog solver reaches 100% accuracy given gold facts, while BERT-style models sit at roughly 50–55% on entailment. Its follow-up (Factoring Statutory Reasoning as Language Understanding Challenges) breaks the task into subtasks, including "argument instantiation", which means filling statute variables with values from the case. It releases SARA v2 with those annotations and Prolog-generated silver data. We reuse the v2 annotations as a direct measure of parse faithfulness.

Can GPT-3 Perform Statutory Reasoning? reports 71% on SARA. It also introduces Synthetic Statutes, clean artificial rules of controlled width and depth, on which two-shot accuracy falls to 81% at width 3 and depth 3. These two findings motivate putting the reasoning in an engine rather than in the model.

The closest system is Language Models and Logic Programs for Trustworthy Financial Reasoning. It uses a frozen large model as the parser into SWI-Prolog, with structure-aware exemplar retrieval and agreement between two parsing runs, and reports over 80% correct on SARA. Our design differs from it in three ways: we update a small model's weights from solver feedback, we treat fact faithfulness as a training objective rather than a hoped-for side effect, and we measure laundering explicitly.

Two further lines supply methods. Execution-based rewards for specialized small models come from Enhancing Language Model with Unit Test Techniques for Efficient Regular Expression Generation, which takes a 13B model from 0.41 to 0.53 pass rate on NL-RX-ST, and from AutoTriton, which uses Group Relative Policy Optimization (GRPO) with rule and execution rewards. Both domains have checkers whose pass/fail signal cannot be faked. A legal parse is different: the engine verifies that the answer follows from the stated facts, not that the facts are true of the case. That difference is the gap this proposal addresses.

JPO is the nearest legal analogue. It trains 3B–8B models with a composite reward over a four-step legal chain and reaches Article F1 90.8 on CAIL2018. Its consistency reward is a statistical proxy (Naive Bayes and co-occurrence), and its authors note that the rationale may still be a post-hoc rationalization. We replace the proxy with execution, and we target rationalization directly.

The accountability literature constrains the design. Fairwashing: the risk of rationalization shows that a surrogate with 90.8% fidelity can hide bias, so high agreement with outcomes is not evidence of honesty. That is why we never score parses by outcome alone. On the Hardness of Computing Counterfactual and Semifactual Explanations in XAI shows that optimal counterfactual search is NP-hard for common model classes. We avoid that search by generating edits on the input side and executing them. The choice of 3B as the smallest model follows OmniCompliance-100K, which reports that models below 1.5B collapse on rule-grounded compliance while Qwen2.5-3B-Instruct holds up.

Core Claim

Let $x$ be a case text and $\pi_\theta$ the parser, which emits a fact program $p = \pi_\theta(x)$. Let $S$ be the statute program and $\mathrm{run}(S, p)$ the engine's answer, with gold label $y$. An edit operator $e$ rewrites one span of $x$ into $e(x)$ and declares the expected fact-level change $\delta_e$, for example "salary of Alice: 45000 → 52000". The training reward is

$$R(x) = \mathbb{1}[\mathrm{run}(S,p)=y] + \lambda \, \mathbb{E}_{e}\big[\mathbb{1}[\pi_\theta(e(x)) = p \oplus \delta_e]\big] - \mu \, \mathbb{1}[p \text{ fails to compile}],$$

where $p \oplus \delta_e$ means $p$ with exactly the declared fact changed, and $\lambda$ and $\mu$ are weights.

The equivariance term needs no gold parse and no gold label for the edited case. It holds the model to one standard of reading across both texts. A model that fabricates facts to hit $y$ tends to fail this term, because fabricated facts do not move coherently under edits it did not anticipate. We define a laundered-correct answer as one where $\mathrm{run}(S,p)=y$ but $\mathrm{run}(S,p_{\text{gold}}^{\text{cf}}) \neq \mathrm{run}(S,p^{\text{cf}})$ on some answer-relevant held-out edit. Equivalently, the answer is right but the reasoning would not survive contestation. Where SARA v2 annotations exist, we also report $p$ against gold arguments.

Research Program

The edit generator is rule-based so that $\delta_e$ is known exactly. It covers numeric amounts, dates and years, names and roles in relationships (spouse, dependent, employer), and boolean facts written as negatable clauses. It produces both answer-relevant edits, which change the gold verdict under $S$, and answer-irrelevant edits, which should change the fact but not the verdict. The irrelevant edits catch a parser that ignores facts it considers unimportant.

We use statute programs in two regimes. The first uses gold Prolog statutes, isolating fact parsing as the paper above does. The second has the model also translate Synthetic Statutes, whose translation can be checked by executing it on generated cases.

Training proceeds in two stages. First, supervised fine-tuning on SARA v2 silver data and generated Synthetic Statute cases. Second, GRPO with the reward above, on Qwen2.5-3B and a 7B model. Training uses the SARA train split plus unlimited synthetic cases, and equivariance edits are applied to both.

We will also analyze where laundering appears during answer-only RL: which fact types get fabricated and whether laundering grows with training steps. This tests whether the problem is real before we credit the fix.

Baselines

The naive reference is the SFT-only parser plus Prolog, trained on silver data with no RL and given the same base model and data. It tells us what execution reward adds at all.

The strongest existing method is the frozen-large-model parser with exemplar retrieval and two-run self-consistency from Language Models and Logic Programs for Trustworthy Financial Reasoning, which reports over 80% correct answers on SARA. We will rerun it with the same gold statute programs, the same test split, and a frontier model available to us. We record cost per case, and we score its parses for faithfulness and laundering with our metrics, which that paper does not report. Beating it means one of two things. Either our 3B or 7B model matches its SARA accuracy within a few points while showing a lower laundered-correct rate at a fraction of inference cost, or it exceeds it on accuracy outright. We also quote the prior reference points from the file: GPT-3 at "up to 71%" and BERT-based models at "around 50-55%" entailment (SARA, GPT-3 statutory reasoning). The 100% hand-crafted Prolog result with gold facts is the ceiling.

The key ablation is answer-only GRPO: the same model, data, compute, and number of policy updates, with $\lambda = 0$. Its purpose is to show whether equivariance, rather than RL itself, is responsible for lower laundering. A second ablation keeps $\lambda > 0$ but uses only answer-irrelevant edits, testing whether relevant edits matter.

We also include inference-time best-of-$n$ with the solver. This samples $n$ parses from the SFT model and keeps the majority answer, with $n$ chosen to match RL training compute amortized over the test set.

All learned variants get the same hyperparameter search budget, tuned on a held-out SARA dev split and never on test.

Why not just…?

Why not just prompt a frontier model as the parser, as the neuro-symbolic tax paper does? This is the real competitor, and on raw SARA accuracy it may win. The evidence against relying on it is indirect. GPT-3 cites the exact Title and Section in only 13% of cases and over-applies rules (Can GPT-3 Perform Statutory Reasoning?), which suggests fluent but unfaithful reading. Self-consistency across two runs also does not test faithfulness, because two runs can agree on the same fabricated fact. Our experiments settle the question by measuring laundering and equivariance on the frontier parser directly. If it already launders rarely, our contribution narrows to cost and on-premise deployment, and we will say so.

Why not just fine-tune on gold or silver parses and skip RL? When gold parses exist, supervised training targets faithfulness directly. For new statutes, however, gold parses are the expensive manual bottleneck the field already notes. Silver data from Prolog is limited to what the generator covers, and the SARA v2 paper reports that argument instantiation lags for numerical and string arguments. The SFT baseline measures this threat. If SFT matches our method on fact-level F1 and laundering, the equivariance reward is unnecessary, and we would redirect the work toward label-free adaptation to new statutes.

Why not just reward correct answers and trust the engine? The engine guarantees that the answer follows from the program, not that the program describes the case. Fairwashing results (Fairwashing: the risk of rationalization) show that optimizing for outcome agreement can produce explanations that look right and are not. The answer-only GRPO ablation tests this. If it shows no laundering increase over SFT, the central motivation fails, and that negative result is itself publishable.

Proposed Experiments

The main evaluation uses SARA entailment and numerical questions on a fixed held-out split. We report answer accuracy, fact-level accuracy against SARA v2 argument-instantiation annotations, laundered-correct rate on held-out edits of types never seen in training, and equivariance accuracy. We separately evaluate on Synthetic Statutes at widths and depths beyond training (3-wide/3-deep and deeper), where GPT-3 two-shot reached 81% (Can GPT-3 Perform Statutory Reasoning?). There we also test the statute-translation regime. To check the narrowing effect documented by Specializing Smaller Language Models towards Multi-Step Reasoning, we evaluate on held-out SARA sections.

Three outcomes decide the proposal. First, answer-only GRPO must raise accuracy over SFT while raising the laundered-correct rate; without that, the problem is not real. Second, the equivariance reward must cut the laundered-correct rate substantially at accuracy within a small margin of answer-only GRPO, and the gain must hold on edit types withheld from training rather than only on those the model was trained against. Third, the 3B model must approach the frozen frontier parser's reported over-80% SARA accuracy at lower cost and with fewer laundered answers. If the second outcome holds only on training edit types, the model has learned to satisfy the edit generator rather than to read faithfully, and we will report that as the limit of the approach.