Diagnosing Hidden Legal Assumptions with Abductive Solvers

When a law student reaches the wrong answer on a problem, the answer is usually not random. It follows from a premise the student believes and never wrote down. Very often that premise is a rule that is true somewhere else: the theft threshold from another code, a common-law element carried into a statutory system, or an older version of the rule. A good tutor finds that unstated premise and asks about it. Current AI tutors cannot do this reliably. Language models are weakest exactly where the reasoning depends on unwritten assumptions and on several statutory conditions holding at once (Why General Language Models Fail at Expert Tasks, Neurosymbolic Language Reasoning as Satisfiability Modulo Theory). We propose to find the student's hidden premise by computing it with a solver.

The approach works as follows. We encode the relevant statutes of the jurisdiction being taught as logical rules. When a student's conclusion does not follow from those rules and the case facts, we ask an SMT solver (an automated checker for logical formulas with arithmetic) to abduce, meaning to search for the smallest extra premise that would make the student's conclusion follow. The search draws its candidates from the rules of other jurisdictions and from nearby threshold values. The premise it returns is a hypothesis about the misconception. The tutor turns it into a plain-language question for the student, such as "were you assuming the amount only needs to exceed X?" The student's reply confirms or rejects the diagnosis, and the question itself serves as the hint. Cross-jurisdictional difference, usually treated as a reason legal models fail, becomes the source of the candidate misconceptions.

Core Claim

Our claim is that abductive diagnosis of student legal reasoning identifies the student's actual misconception more accurately, and yields hints that give away less of the answer, than two alternatives: a frontier language model given the same statutes, and a solver pipeline that only reports which step is invalid. We expect the advantage to be largest on offenses with numeric thresholds and several conjoined elements. The tutor should also extend to a new jurisdiction by adding a rule encoding, with no retraining. Stated precisely, let $R_J$ be the encoded rules of the jurisdiction being taught, $F$ the facts of the case, and $c$ the student's conclusion, where $R_J \wedge F \not\models c$. We seek

$$\phi^* = \arg\min_{\phi \in G(R_{J'}, \Theta)} |\phi| \quad \text{s.t.} \quad R_J \wedge F \wedge \phi \models c, \;\; R_J \wedge F \wedge \phi \not\models \bot .$$

Here $G(R_{J'}, \Theta)$ is a grammar of candidate premises. It is built from the predicates shared across jurisdictions, from rules $R_{J'}$ taken from other jurisdictions, and from a set $\Theta$ of alternative numeric thresholds. The term $|\phi|$ is the syntactic size of the premise, and the second condition requires the premise to be consistent with the rules and facts. Candidates that exactly match a rule in some $R_{J'}$ receive a ranking bonus, because they correspond to recognizable transfer errors.

Relation to Prior Work

The mechanism comes from interactive verification. An Interactive SMT Tactic in Coq using Abductive Reasoning couples Coq with cvc5's syntax-guided synthesis to propose the missing lemma when a proof stalls. On the Zorder library it resolved 13 of 26 goals that direct SMT calls failed, and calls finished in under 8 seconds, fast enough for dialogue. Talking with the Theorem Prover to Interactively Solve Natural Language Inference turns unproven subgoals into English questions for people, raising recall on SICK from 32.1% to 44.1%. We reuse both techniques but reverse their direction. In that prior work the missing premise is knowledge the prover lacks and a human supplies. In ours it is an error the learner holds, and the solver infers it. Sumita et al. also constrain our design. Their 40% human agreement on proposed axioms warns that asking non-experts to judge axioms is noisy. We therefore ask the student about their own belief, which is a question about the student rather than about the law.

The closest tutoring system is LeanTutor. It translates student proof steps into Lean, compares them with a staff reference proof, and writes Socratic hints. On PeanoBench it cut answer leakage from 2.2 to 1.1 and raised next-step hint accuracy from 2.8 to 3.7 out of 5 against GPT-4o-mini. LeanTutor diagnoses by divergence from a reference solution, which tells the student where they left the path but not what they believe. It also works only in Peano arithmetic. In law, formal machinery has so far been built for adjudication rather than teaching. Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning autoformalizes Chinese criminal statutes into Horn clauses for Z3 and uses adversarial agents to extract facts. VeriCoT checks each chain-of-thought step by entailment, including on US tax statutes. We reuse L4L's statute encoding and fact-tuple format. Its known failure, where missing facts in the narrative break the solver, is the main risk we must control. VeriCoT is our verify-only competitor: it can say a step fails, but it does not say which belief produced the failure.

On the jurisdiction side, LegalReasoner trains on 58,130 Hong Kong cases (LegalHK) and tests on mainland Chinese cases. It gives us a documented pair of jurisdictions where transfer is known to be hard. ProMALex handles jurisdictional variation with hierarchical adapters, and LEEC shows that legal element definitions do not map across systems. We deliberately put jurisdictional knowledge into swappable rule sets rather than weights. LawBench (the fine-tuning tax) and the L'ART study (the legal bias paradox) both suggest that folding law into the weights degrades reasoning. From Solver to Tutor (KMP-Bench) shows that specializing a model for solving also makes it a worse tutor.

Research Program

The build has three parts.

The first is a rule base covering a deliberately small slice of criminal law: theft, fraud, and drug possession. Each offense is encoded for mainland China, starting from L4L's encoding over LeCaRDv2 and LEEC, and for Hong Kong. For Hong Kong the drafts are produced by a language model and every rule is signed off by a lawyer. These two jurisdictions differ in exactly the ways the cross-jurisdiction literature flags: monetary thresholds, element definitions, and deductive versus precedent-driven structure (Cross-Jurisdictional Generalizability in Legal AI). A third encoding for one held-out offense is reserved to test extension by rule addition alone.

The second part is the diagnosis loop. A model extracts fact tuples and the student's stated conclusion and intermediate claims. The student can see and correct the extracted facts, which is our guard against garbage in, garbage out. cvc5 then runs abduction over the grammar defined above and returns ranked candidate premises. A hint writer renders the top candidate as a confirmation question under a no-leak constraint: the question may name the suspected belief but may not state the correct rule or the verdict. If the student rejects the candidate, the loop moves to the next one. If all candidates are rejected, the system falls back to step-level feedback.

The third part is a benchmark, because no student-error dataset for legal reasoning exists in the material. We build it in two layers. The first layer contains seeded errors. We take cases from LeCaRDv2 and LegalHK and generate wrong student analyses, each produced by substituting one known premise, such as a foreign rule, a wrong threshold, or a dropped element. The seeded premise is the ground-truth misconception, in the style of PeanoBench's incorrect proofs. The second layer is a smaller set of genuine answers from law students in both jurisdictions. For each answer, two instructors annotate the misconception, and we report their agreement.

Baselines

The naive reference is a general frontier model acting as a tutor with no statutes supplied, prompted to diagnose the error and give a hint. This shows what fluent general knowledge alone achieves.

The strongest competitor is the same frontier model given the full encoded-source statute text for both jurisdictions and the extracted facts. It is asked with chain-of-thought prompting to name the student's hidden assumption and then hint. It is the strongest because it has every piece of information the solver has, and frontier models are good at pattern-matching well-known errors. It runs on identical cases and the same fact extraction, with a matched prompt-tuning budget on a shared development split. We allow it several samples and self-consistency voting, so that it spends at least as much compute as our pipeline. There is no reported result on Lacuna for misconception diagnosis in legal tutoring. The nearest reference points are these: DeepSeek-V3 reaches 73.1% overall on KMP-Bench's math tutoring evaluation. On AR-BENCH, Qwen3-Max detects judgment errors at 92.03% but corrects them at only 74.10%, and on a 150-case subset it reaches 53.8% versus 70.1% for humans (AR-BENCH). That gap between detecting and correcting is exactly the kind of gap we expect between "something is wrong" and "here is what you believed." Beating this baseline means a statistically significant gain in top-1 misconception identification on real student answers, with no higher leakage.

The verify-only competitor is VeriCoT-style checking. The same rule base and solver identify the first invalid step, and a language model explains it. This isolates whether abduction adds anything beyond formal step checking.

A LeanTutor-style competitor diffs the student's steps against a formalized reference analysis.

Two ablations isolate the key ingredients. The first removes other-jurisdiction rules from the grammar, leaving only thresholds and dropped elements. This tests whether cross-jurisdictional candidates drive the gains. The second removes the student confirmation step and delivers the top candidate as a statement. This measures what the human-in-the-loop turn contributes.

Why not just…?

Why not just give a frontier model both statutes and ask it what the student assumed? This is the real threat, and on single-element errors it may well tie. The evidence that it falls short concerns conjunctions and arithmetic. Multi-step LegalBench tasks such as successor liability sit near or below 50% even for GPT-4 (LegalBench). L4L's sentencing error drops from 32.84 months RMSE for GPT-4o to 12.72 months when a solver is added. On AR-BENCH, supplying full statute text hurt diagnosis compared with supplying the reasoning alone. Our seeded benchmark stratifies errors by the number of conjoined elements and by whether a threshold is involved. If the language-model baseline matches us in the complex strata too, the solver is unnecessary and we will report that.

Why not just verify each step and let the model explain the failure? Step verification locates an error but leaves the model to guess the reason. KMP-Bench finds that flawed scaffolding (32.5% of failures) and substituting direct hints for guidance (25.8%) are the dominant tutor failures, and a vague reason for an error invites both. The verify-only baseline answers this question directly. If it matches abduction on identification accuracy and leakage, the abduction step is not earning its complexity.

Why not just fine-tune a tutor on legal dialogues or jurisdictional adapters? Two findings argue against it: the fine-tuning tax (LawBench) and the legal bias paradox, where Legal BERT scores 56.19% versus 61.62% for BERT Base on L'ART. Adapters also need data for every new jurisdiction. We test this on the held-out offense: the rule-based tutor gets only a new encoding, while a fine-tuned comparison model gets an equivalent annotation budget.

Why not drop the human confirmation and diagnose automatically? Automatic diagnosis is cheaper and avoids dialogue latency. The confirmation question costs the student little and is itself a Socratic move, which matters because real students bypass effortful scaffolding (Rethinking Scaffolding in LLM Tutors). The second ablation measures whether confirmation improves final diagnosis accuracy and student uptake.

Proposed Experiments

On the seeded benchmark, the primary metric is top-1 and top-3 misconception identification, reported separately for errors involving foreign rules, thresholds, and dropped elements. Each result is reported both with gold facts and with extracted facts, which quantifies the extraction risk L4L leaves open. On the real student answers, we score identification against instructor labels. We score hints on the KMP-Bench dimensions of mistake identification, mistake location, guidance, and actionability. Leakage is scored on LeanTutor's 1–5 scale using a rubric-based language-model judge, which we validate against instructor ratings before relying on it, following EduMod-LLM.

The decisive checks are three. First, the abductive tutor must beat the full-statute frontier baseline on real student answers in the multi-element and threshold strata. Second, removing cross-jurisdiction rules from the grammar must measurably reduce accuracy on transfer errors. If it does not, the jurisdictional framing is decoration. Third, on the held-out offense, adding a rule encoding alone must match the other offenses' accuracy within the confidence interval.

A small classroom pilot then compares, for each of confirmation questions and direct-statement hints, whether students revise their analysis correctly in the next turn. This addresses whether benchmark gains survive contact with students who would rather just be told the answer.