Federating Faithfulness Across Legal Jurisdictions

Legal AI systems invent things. Commercial tools that retrieve real case law still attribute made-up holdings to genuine cases in 17% to 33% of queries (Hallucination-Free?). The expert annotations that could teach a model not to do this are among the most expensive data in machine learning. CUAD's contract labels alone cost an estimated $2M of attorney-supervised labor (CUAD). Those labels are also locked to one legal system and often to one institution's private records. Each country, court system or firm can therefore afford only a small pile of expert-verified examples of hallucination. No one can legally or practically pool those piles.

This note proposes a way to share what those piles have in common without sharing the law they contain. The key is to split what a legal model learns into two parts. The first is jurisdiction-specific content, such as which theft amount triggers which sentence. The second is a jurisdiction-agnostic behavior: staying faithful to the source text in front of it and not substituting a memorized number, date, party or obligation. Only the second part is trained jointly across jurisdictions, using federated learning (clients train on their own data and send only model updates to a coordinating server). The content stays in per-jurisdiction modules. The central empirical question is whether expert-verified faithfulness signal from one legal system reduces hallucination in another, and whether the federated setup can deliver that benefit without making the model mix up whose law applies.

Core Claim

We claim that faithfulness can be made transferable across legal systems by construction, and that the construction is simple. Each jurisdiction's experts label source-conditioned preference pairs. In each pair, the relevant local source (a statute, a clause, a judgment passage) is in the prompt. The preferred answer agrees with that source. The rejected answer contains a typed deviation, either a natural model error or an expert-verified perturbation: a wrong numeric cap, a wrong period, an inverted obligation, or a swapped party.

Because the correct answer is always recoverable from the context, these pairs carry no answer that is specific to one jurisdiction. Training a shared low-rank adapter on them should teach "do not override the source" rather than "China's threshold is X." Content pairs, where the correct answer depends on memorized local law, update only a local adapter that never leaves the client. (A low-rank adapter, or LoRA, is a small add-on weight matrix fine-tuned while the base model stays frozen.)

The novel prediction is two-sided. First, a jurisdiction held out entirely from training should see its hallucination rate fall when other jurisdictions' faithfulness pairs are shared. Second, naive federation of all preference data should raise a new, measurable error: citing another jurisdiction's rule with full confidence. We call this jurisdictional contamination. We will show that the proposed split suppresses it.

Relation to Prior Work

Three lines of work each hold one piece of this problem.

Federated legal learning shows that decentralized training is feasible and fragile, and we build on it for method. FedJudge federates LoRA adapters for a 7B model across heterogeneous legal tasks. It adds a constraint to limit forgetting, and in one setting it beats centralized pooling (Legal Consultation BertScore 64.33 vs 62.27). FEDLEGAL shows the opposite on naturally split court data, where FedOPT reaches 19.30 Macro-F1 against 39.90 centralized. Both stay inside Chinese law, and neither measures hallucination. We take their communication design and their warning about heterogeneity, and we add a cross-border setting and a factuality objective.

Cross-jurisdictional modeling motivates the split between shared and local parameters, and it constrains how we build it. ProMALex clusters jurisdictions by embedding similarity. It shares lower-layer adapters and branches upper-layer ones, improving multilingual perplexity to 5.468 from 7.104. However, it is centralized and evaluated only on perplexity. Beyond Borders uses adversarial training to make representations jurisdiction-invariant. It finds that this erases specialized legal knowledge in Legal-Pegasus. We read that negative result as ruling out forcing all knowledge to be invariant. We make only the faithfulness signal shared and leave content explicitly local, which is ProMALex's philosophy applied to the objective rather than to the architecture alone.

Legal hallucination work supplies the objective and the error taxonomy. HIPO adds a likelihood term on persistent hard errors to direct preference optimization (DPO, which trains a model to prefer one response over another from labeled pairs). It forces verbatim statute recitation, reaching 38.35% non-hallucinated statute rate on LegalHalBench, but only centrally and on Chinese civil law. LegalHalluLens supplies the four-type taxonomy: numeric, temporal, obligation and factual. It shows that aggregate rates hide a roughly 40-point gap between types, and that a six-role debate at inference time cuts fabrications by 45%. The authors say this approach filters errors but does not repair them. We reuse HIPO's loss and LegalHalluLens's taxonomy as the shared annotation schema across clients. The taxonomy is what makes experts in different legal systems label the same kind of thing.

Research Program

The system has one frozen base model and two kinds of LoRA adapters. The first is a shared faithfulness adapter aggregated across clients. The second is one local content adapter per jurisdiction, never transmitted. Each round, a client computes a HIPO-style loss on its source-conditioned pairs and sends only the shared-adapter update. Content pairs train the local adapter alone. Writing $x$ for the prompt with its source text, $y^{+}$ and $y^{-}$ for preferred and rejected responses, and $\theta_s$ for the shared adapter, each client $k$ minimizes

$$ \mathcal{L}_k(\theta_s) = \mathbb{E}_{(x,y^{+},y^{-}) \in F_k}\big[\ell_{\mathrm{DPO}}(x,y^{+},y^{-}) - \lambda \log p(y^{+}\mid x)\big]. $$

Here $F_k$ is client $k$'s set of expert-verified faithfulness pairs, $\ell_{\mathrm{DPO}}$ is the standard preference loss, and $\lambda$ weights the likelihood term that HIPO found necessary for exact recitation. Hard-sample filtering follows HIPO: pairs the current model still gets wrong are replayed.

The annotation protocol is part of the contribution, and it follows the concept-first design of Ni et al.. Experts first fix which source span governs the answer, then judge model outputs against it by error type. This keeps the per-example cost low enough to sweep annotation budgets.

Four jurisdictions form the clients.

  • China (statutes): LegalHalBench-style statutory question answering.
  • United States (contracts): CUAD clauses under the LegalHalluLens protocol.
  • United Kingdom and India (judgments): abstractive summarization on UK-Abs and IN-Abs, with the text of the source judgment as context.

Summarization with a source judgment in context is exactly where fabricated holdings arise. A separate probe set tests contamination. Experts write questions whose answers genuinely differ across jurisdictions, such as limitation periods, monetary thresholds and who counts as the victim in card fraud (Chen; Gan et al.). We count how often the model gives a rule from a jurisdiction other than the one asked about.

Privacy leakage through updates (FEDLEGAL reconstructs text from gradients) is real but orthogonal. We will report results with and without secure aggregation, and we do not claim a privacy contribution.

Baselines

The naive reference point is the base model with retrieval of local sources and no fine-tuning. For US legal queries, the closest published figure is that GPT-4 hallucinated 43% of the time on the pre-registered queries of Hallucination-Free?, while the best commercial retrieval tool, Lexis+ AI, hallucinated 17%. We cannot run the commercial tools. We instead run our own base model with the same retrieved sources, as the "do nothing" floor.

The strongest single-jurisdiction competitor is centralized HIPO trained separately in each jurisdiction on that jurisdiction's expert pairs. On LegalHalBench it reports 38.35% non-hallucinated statute rate on GLM4-9B, against 16.0% for GPT-4o (HIPO). We run it with the same base model, the same expert pairs, the same adapter rank and the same number of hyperparameter trials as our method. Beating it means lower typed hallucination rates at equal expert budget, especially for the lowest-budget jurisdiction, without losing task quality.

The strongest inference-time competitor is the calibrated debate pipeline of LegalHalluLens, which reports a 45% reduction in fabricated extractions on CUAD. We run it on the same base model and report its hallucination rates alongside its roughly sixfold inference cost. Beating it means matching its fabrication reduction on CUAD at single-pass cost. We also test whether stacking it on our model adds further gains, which would indicate the two are complementary rather than competing.

Two federated baselines test the structure. The first is the federated LoRA recipe of FedJudge, extended with a HIPO loss over all preference pairs and a single shared adapter. The second is a centralized oracle that pools every client's pairs, as an upper reference that is illegal in practice. We report no reported result found on Lacuna for hallucination rates of federated legal models, or for faithfulness of generated summaries on UK-Abs, IN-Abs or BVA-Ext. Those rows are new measurements.

The key ablation removes the content/faithfulness routing and sends all pairs through the shared adapter. If contamination does not rise under this ablation, the split is unnecessary.

Why not just…?

Why not just train each jurisdiction's model locally and skip federation? This is the most serious threat. Hallucination is partly about local knowledge, and local training has no heterogeneity penalty of the kind FEDLEGAL documents. Our bet is that faithfulness to the provided source is a behavior that scarce expert labels teach slowly, so pooling it helps jurisdictions with small budgets. The budget sweep below tests this directly. If per-jurisdiction HIPO matches the federated shared adapter at every budget, we will report that federation of expert labels is not worth its complexity for this goal.

Why not just generate typed perturbations synthetically from public statutes and case law, train centrally, and drop both experts and federation? Statutes are public, so this is a genuine threat. Our reason to doubt it is that the hard errors come from the model's own priors rather than from random corruption. LegalHalluLens finds models inserting a stereotyped $5M liability cap even when told not to. CUAD finds that unlabeled legal text bought only about 3% while expert labels moved precision from 8.2% to 44%. We include a synthetic-only centralized arm trained with the same loss and matched pair counts. If it matches the expert-verified arm on natural error rates, expert annotation is not the bottleneck here, and we will say so.

Why not just rely on retrieval plus inference-time verification? Retrieval is kept in every arm, since all our faithfulness pairs include the source. The question is whether the residual unfaithfulness to retrieved text can be trained away. Hallucination-Free? shows that residual is large even in the best commercial systems. LegalHalluLens and AR-BENCH, where correction lags detection at 74.10% vs 92.03%, both indicate that verification filters errors better than it repairs them. The debate baseline measures how much training-time faithfulness adds beyond that.

Proposed Experiments

The first decisive check is a leave-one-jurisdiction-out test. We train the shared adapter on three clients and evaluate on the fourth, with no local data for the held-out client. We measure typed hallucination rates, the Risk Direction Index from LegalHalluLens (whether errors lean toward invention or omission), and non-hallucinated statute rate on the Chinese split. For the UK and India, we measure groundedness, meaning whether cited authority supports each claim, following Hallucination-Free?. The prediction is a drop in numeric and obligation errors on the held-out jurisdiction relative to the untuned base. These are the categories LegalHalluLens finds near 70% error. No gain would falsify the transferability claim.

The second check is contamination. We compare the routed design against the all-pairs ablation and against the pooled oracle on the cross-jurisdiction probe set. The prediction is that the all-pairs federated and pooled models cite foreign rules more often than the base model, and the routed design does not.

The third check is an expert-budget sweep. We cut one jurisdiction's expert pairs to a tenth of the others' and plot its hallucination rate under local-only HIPO, federated shared adapter, and synthetic-only training. The proposal succeeds if federation closes most of the gap to the full-budget local model where local-only training cannot.

Task quality is tracked throughout, with ROUGE-L and BERTScore on the summarization corpora (baseline 0.546/0.839 on the BVA-Ext to UK-Abs setting reported by Beyond Borders) and CUAD extraction precision. The purpose is to confirm that reduced hallucination is not bought with more abstention or omission, and the Risk Direction Index will expose that failure if it occurs.