When a court rules against a plaintiff in a negligence appeal, it usually does so by pointing to a few specific facts. The defendant posted a warning sign. The hazard was open and obvious. The plaintiff was trespassing. Change one of those facts and a competent lawyer would expect the ruling to change too. Change an irrelevant detail, such as the weekday or the plaintiff's occupation, and the ruling should stay put. This proposal builds a test around that intuition. Lawyers mark the "pivot facts" in real appellate opinions and then write matched pairs of edits. One edit changes a pivot fact and should flip or weaken the outcome. The other changes a salient but legally inert fact and should change nothing. A legal prediction model reasons transparently only to the extent that its verdicts move under the first kind of edit and hold still under the second.
The opportunity exists because current transparency measures for legal AI agree poorly with each other and with experts. Token-highlighting explanations agree with expert rationales at roughly chance level. Generated explanations hallucinate. Models with higher accuracy sometimes track judicial reasoning less well. The moral reasoning literature already has a sharper tool for exactly this problem: defeasibility. Defeasibility is the property that a judgment can be overturned by one added fact, and that literature measures it by asking whether added context strengthens or weakens a verdict. That tool has only been applied to crowdworker vignettes. This note moves it into case law, with domain experts supplying the counterfactuals. It also tests one specific claim: training on expert counterfactual pairs makes models more sensitive to pivot facts in ways that ordinary accuracy cannot see.
Core Claim
Our central claim is that counterfactual sensitivity to expert-identified pivot facts is a better measure of legal reasoning than accuracy, rationale overlap, or saliency. Two predictions follow. First, models with similar outcome F1 on appellate negligence cases will differ widely in pivot sensitivity. Second, a modest number of expert-written counterfactual pairs, used as training signal, will raise pivot sensitivity on held-out cases more than the same number of LLM-generated pairs. We define pivot sensitivity for a model $f$ as
$$\Delta(f) = \Pr[f(x') \neq f(x) \mid \text{pivot edit}] - \Pr[f(x') \neq f(x) \mid \text{inert edit}]$$
Here $x$ is the original case facts and $x'$ is an edited version. The first term is the flip rate under expert pivot edits, and the second is the flip rate under the matched inert edits. Subtracting the inert term penalizes models that flip on any perturbation. We also report a soft version for models that output a distribution: the divergence between original and edited verdict distributions, in the spirit of the Jensen–Shannon defeasibility reward of ClarifyDelphi. For strengthening edits we separately score whether the shift goes in the expert-specified direction.
Where This Sits
Three strands of prior work motivate the gap.
On the Role of Negative Precedent in Legal Outcome Prediction shows that European Court of Human Rights (ECtHR) outcome models score 10.09 F1 on rejected claims, worse than random. A claim–outcome structure raises this to 24.01. That fix is purely structural and says nothing about which facts defeat a claim. Towards Explainability in Legal Outcome Prediction Models traces model predictions back to training cases using influence functions, a method that estimates how much each training case shaped a given prediction. It finds that this influence correlates negatively with precedents judges cite in order to distinguish them, and identifies an "F1 trap": higher accuracy does not mean closer alignment with judicial reasoning. Both papers diagnose a failure at distinguishing. Neither supplies case-level annotations of the facts that do the distinguishing, and that is what we collect.
On the moral side, ClarifyDelphi supplies the method we reuse: score context by how far a strengthener or weakener shifts a verdict. It reports that answers to its questions flip the Delphi model's judgment 25% of the time, versus 21% for a fine-tuned T5 model. Its strengtheners, its weakeners, and its judge are all products of crowdworkers or Delphi, so they carry no legal standard. We replace both the edits and the ground-truth direction with expert judgment, anchored in what a real court relied on.
AnnoCaseLaw supplies the substrate and a constraint. It provides 471 U.S. appellate negligence cases with law-student annotations of facts, precedents, 36 legal concepts, and the court's application of law to facts. It also reports that frontier LLMs sit at 0.45–0.50 F1 from facts alone and reach about 0.70 only when given the court's own application section. Supplying precedents barely helps. The jump to 0.70 is ambiguous: a model may reason better, or it may simply read the answer out of the court's analysis. Pivot edits separate these two explanations.
The constraint side also matters. VECHR finds that Integrated Gradients saliency agrees with expert highlights at κ≈0.10. PredEx finds that generated rationales are fluent, but expert raters give them only 2.84–3.30 out of 5, and the rationales are often inconsistent with the model's own verdict. Together these rule out evaluating transparency by asking the model to explain itself. The neuro-symbolic moral judgment work shows that unguided chain-of-thought can over-focus on narrow features, while structured feature extraction raises MoralExceptQA F1 from 64.47 to 84.34. That result motivates the structured-feature model in our comparison.
The closest evaluation design is LJPCheck, which proposes functional tests for legal judgment prediction. The file gives no detail on its tests or results, so we cannot compare numbers. As we understand functional testing, it relies on generic, templated perturbations. Our contribution differs in three ways: the edits are written for each case by experts, they are anchored to the facts a real court used to distinguish precedent, and each comes with a directional label and a matched inert control.
Research Program
The first component is an annotation protocol borrowed from the expert-annotation literature. ClimRetrieve uses a "concept-first" workflow in which experts define concepts before labeling. HBCP pairs an explicit taxonomy with double annotation and reconciliation and reaches α=0.74. HBCP also shows that ill-defined targets can fall to α=−0.14, a warning about loose label definitions.
We therefore start by having a supervising attorney write operational definitions for three things. A pivot fact is one the court explicitly relied on in applying a negligence element, such as duty, breach, causation, or a defense like contributory negligence, which are AnnoCaseLaw concept categories. A pivot edit is a minimal change to such a fact that would, under the court's own stated reasoning, defeat or reinforce that element. An inert edit is a change of similar length and salience to a fact the court never relied on.
Final-year law students then work on roughly 200 AnnoCaseLaw cases, drawn so that affirmances and reversals are balanced. For each case they mark two or three pivot facts and write one pivot edit and one inert edit per pivot fact. A second annotator independently predicts the edited outcome without seeing the first annotator's label. Only pairs where both annotators agree enter the evaluation set, with an attorney adjudicating borderline cases, following CUAD's verification practice. Pairs where the experts disagree are kept separately as a distribution of judgments, which follows SCRUPLES's treatment of disagreement as signal rather than noise. We will report agreement per concept. Low agreement on, say, duty of care is itself a finding.
The second component is the diagnostic. We run every model under the same three context conditions AnnoCaseLaw uses. Edits always modify the facts section, and edited cases never include the court's application section. This forces the model to reason from the facts rather than copy the court's conclusion. Because the original opinions are public and may be memorized, edits also act as a contamination probe: a model that recalls the outcome will fail to flip.
The third component is an intervention, and it isolates the value of expert annotation. Half the cases provide training pairs and half are held out. We fine-tune an open model on outcome labels alone, on outcome labels plus expert counterfactual pairs, and on outcome labels plus an equal number of LLM-generated pairs. We also build a structured model in the style of the neuro-symbolic moral work. An LLM extracts values for the pivot-fact categories defined by the experts, and a transparent classifier predicts the outcome from those values, so its sensitivity can be read off directly.
Baselines
The naive reference points are majority-class prediction and a frontier LLM given facts and procedural history zero-shot. AnnoCaseLaw reports that the latter reaches 0.45–0.50 F1. For pivot sensitivity there is no prior number: we found no reported result on Lacuna for pivot-edit flip rates on legal cases. The naive model defines $\Delta=0$ trivially, and the zero-shot LLM is our empirical floor.
The strongest existing outcome predictor is the frontier LLM given facts plus the court's application-of-law section. AnnoCaseLaw reports about 0.70 F1 for this setup. We run it under identical prompts and model versions, but with the application section withheld from edited inputs, as explained above. Beating it in our sense does not mean exceeding 0.70 F1. It means matching its F1 on originals within a few points while achieving a clearly larger $\Delta$. We set the threshold for "clearly larger" before testing: the lower bound of a bootstrap 95% interval must exceed the baseline's point estimate.
A second strong competitor is precedent-enhanced prediction in the style of PLJP. That system restructures facts into motivation, behavior, and after-the-fact circumstances, then retrieves precedents to guide the prediction, and reports 94.18% accuracy on Chinese criminal cases. We adapt it to negligence using AnnoCaseLaw's precedent annotations. This also tests whether fact-structuring explains the gap between PLJP's success and AnnoCaseLaw's precedent paradox.
For transparency we compare against two existing measures: Integrated Gradients saliency and generated rationales, each scored against AnnoCaseLaw's expert spans. The question is whether those measures rank models the same way $\Delta$ does.
The key ablation compares expert pairs with LLM-generated pairs at equal count, equal compute, and an equal hyperparameter search budget of the same number of trials on the same validation split.
Why not just…?
Why not just generate the counterfactuals with an LLM and skip expert annotation? This is the most serious threat, and the ablation is built to settle it. We will have experts audit a sample of LLM-generated pivot edits for legal validity, meaning whether the edit would actually flip the outcome under the court's stated reasoning, and report that validity rate. Our expectation is that LLM edits flip surface vocabulary rather than the element the court relied on. Two pieces of evidence support this. LLMs perform below random on "Duty of Care" concept identification in AnnoCaseLaw. In CUAD, expert labels moved precision at 80% recall from 8.2% to 44.0%, while 8GB of extra unlabeled legal text added only about 3%. If expert-validated LLM edits match expert-written pairs on held-out $\Delta$, we will conclude that the efficient pipeline is LLM generation followed by expert filtering, and we will report that as the result.
Why not just measure overlap with AnnoCaseLaw's existing rationale spans? Overlap rewards a model for attending to the right text, not for depending on it. VECHR's κ≈0.10 shows that span-based agreement is noisy, and the F1 trap shows that the quantity it is meant to proxy diverges from accuracy. We test this directly by computing the rank correlation between span overlap and $\Delta$ across all models. A high correlation would make the new annotations unnecessary.
Why not just ask the model for a chain-of-thought explanation and check it? Self-explanation measures what the model says, not what changes its verdict. PredEx documents verdict–explanation inconsistency. The neuro-symbolic moral work shows that chain-of-thought can over-focus on narrow features. We include chain-of-thought prompting as a condition and report whether it raises $\Delta$ or only produces more fluent text.
Why not just write expert labeling functions and use weak supervision? BOXWRENCH shows that expert-written labeling functions beat full supervision by 5% on unfair-contract detection. Those tasks are flat classification, though, and a pivot edit depends on how each fact interacts with the specific case. We treat this as a cheaper follow-up rather than a competitor. If held-out $\Delta$ gains concentrate on a few recurring pivot types, such as open-and-obvious hazards or trespasser status, labeling functions become the scalable next step.
Decisive Checks
Three outcomes would confirm the proposal. First, among models within 0.05 F1 of one another on original AnnoCaseLaw cases, $\Delta$ varies substantially, and its rank order does not match the order given by span overlap or saliency agreement. Second, expert-pair training raises held-out $\Delta$ above both outcome-only training and LLM-pair training, without lowering original-case F1 by more than a few points. Third, as a transfer test, a model trained with pivot pairs improves on negative outcomes in the ECtHR negative-precedent data relative to the reported 24.01 F1 of the claim–outcome model, evaluated on the same split. We also run the full diagnostic on a small set of about 60 ECtHR distinguished cases with expert edits.
The result would be negative if no model shows a $\Delta$ meaningfully above zero, or if annotators cannot agree on edited outcomes (α well below HBCP's 0.74). Either result would mean that pivot facts in negligence law are too open-textured to annotate reliably, and we would report that finding.
References
- ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
- On the Role of Negative Precedent in Legal Outcome Prediction
- Towards Explainability in Legal Outcome Prediction Models
- AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction
- VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
- Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts
- Neuro-Symbolic Models of Human Moral Judgment: LLMs as Automatic Feature Extractors
- LJPCheck: Functional Tests for Legal Judgment Prediction
- ClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures
- HBCP Corpus: A New Resource for the Analysis of Behavioural Change Intervention Reports
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
- SCRUPLES: A Corpus of Community Ethical Judgments on 32, 000 Real-Life Anecdotes
- Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration
- Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks