A legal language model trained on one country's courts is often deployed in another. The practitioner then has a model-selection problem: which source jurisdiction, which adaptation method, and which checkpoint to ship, usually with few or no labeled cases from the target jurisdiction. The default label-free signal is perplexity on target-jurisdiction text, meaning how surprised the model is by that text. Perplexity is weak here for two reasons. First, topic-modeling research found long ago that the likelihood-best model is often not the one whose internal groupings make sense to people. Second, public legal corpora are almost certainly in web-scale pre-training data, and a model that has memorized target judgments will show low perplexity without having learned anything that transfers.
We propose a different signal, which we call doctrinal coherence. Take the candidate model's internal representations of unlabeled target-jurisdiction cases and cluster them. Have a fixed language-model labeler name the legal issue of each cluster's most representative cases. Then score how consistently each cluster's cases share one issue. A model that has actually carried legal structure across the border should sort a foreign court's cases into doctrinally clean groups. A model that has only memorized text, or whose legal knowledge was stripped out during adaptation, should not. We test whether this signal picks better models than perplexity, including perplexity computed on text published after the model's training cutoff. We also test whether it resists deliberately planted benchmark contamination.
Core Claim
Our claim is that document-level coherence of a model's target-jurisdiction representations predicts downstream cross-jurisdiction performance better than target perplexity. We expect the advantage to be largest in two situations: when adaptation damages legal knowledge, and when the selection text overlaps with the model's training data.
The measure is adapted from topic-model evaluation. Stammbach et al. used an LLM to label the ten most representative documents of each topic. They showed that the agreement of those labels, which they call label purity, tracked ground-truth clustering quality better than scoring top-word lists when choosing the number of topics. We apply the same measure to a different object. Instead of choosing a topic count within one corpus, we score competing neural legal models on how they organize a foreign jurisdiction's case law.
Formally, for candidate model $m$, let $\mathcal{C}_k(m)$ be the $k$ clusters obtained from $m$'s pooled representations of an unlabeled target-case pool. For each cluster $c$, let $\text{pur}(c)$ be the largest fraction of its representative cases that receive the same issue label from the fixed labeler. The model's score is
$$S(m) = \max_{k \in K} \frac{1}{k} \sum_{c \in \mathcal{C}_k(m)} \text{pur}(c),$$
where $K$ is a small grid of cluster counts. Taking the maximum over $k$ keeps the score from depending on one arbitrary granularity. This matters because coherence scores are known to shift with topic cardinality (The Sensitivity of Topic Coherence Evaluation to Topic Cardinality).
Why this gap is real
Three pieces of prior work motivate the proposal.
Santosh et al. studied zero-shot summarization transfer among UK Supreme Court, Indian Supreme Court and US Board of Veterans' Appeals judgments. They found that domain-adversarial training helps general models. This technique uses a gradient-reversal layer, which trains the encoder to hide which jurisdiction a text came from. On the legally pre-trained Legal-Pegasus, however, it caused "representation erasure" and lowered performance. They also note that ROUGE and BERTScore cannot verify legal correctness.
ProMALex builds a tree of jurisdictions from dataset embedding similarity. It then branches low-rank adapters (small trainable add-on weight matrices) across that tree, and both builds and judges the design by perplexity. Neither paper checks whether the chosen model organizes the target's law coherently. Neither controls for leakage.
On the contamination side, Yao et al. showed that contamination through translated benchmarks evades n-gram and perplexity checks. Their detector only works for multiple-choice formats. Yax et al. showed that memorized items produce near-zero surprise, which is exactly the failure mode of perplexity-based selection.
LeKUBE constrains the design. There, fine-tuned models recite updated statutes at about 93% accuracy but cannot apply them. So any selection signal that rewards reproducing text, rather than organizing it, is suspect from the start.
The literature also contains a tension we must face directly. ProMALex reports that perplexity-driven adapter branching wins on held-out modeling loss. That is not evidence about downstream legal quality. Part of our program is to test whether ProMALex's perplexity ranking survives a downstream check.
Research Program
Candidate pools. We build pools where the true ranking is unknown and the choice matters.
In the common-law summarization testbed, we reproduce the Santosh et al. grid. It crosses BART, BERT-BART and Legal-Pegasus with each source jurisdiction, with {plain fine-tuning, gradient-reversal adaptation, MMR silver-summary augmentation}, and with several training checkpoints. This gives on the order of a hundred candidates per target. The known Legal-Pegasus erasure case serves as a built-in sanity check.
In the multi-jurisdiction testbed, we train ProMALex-style adapter configurations on LexFiles and Multilegalpile, varying branching depth and rank. We evaluate them downstream on Swiss-Judgment-Prediction.
Temporal splits. For every target, we assemble unlabeled pools of cases published before and after the base models' training cutoffs.
For UK and Indian Supreme Court judgments, this means collecting newly released judgments in the same format as UK-Abs and IN-Abs. For the Swiss target, it means newer decisions.
For Chinese criminal law, we use CAIL2018 as the old, public split. The JPO-Dataset, roughly 270k cases from 2024–2026, serves as the naturally post-cutoff split. This Chinese pair is a within-jurisdiction calibration of how much contamination inflates each selection criterion. It is not a transfer test.
Coherence computation. Representations come from each candidate's encoder, or from mean-pooled decoder states for generative models. We use the representative-exemplar variant of Bilal et al. to pick each cluster's central cases. This avoids the quadratic pairwise cost they report.
The labeler is fixed across all candidates. We run both an open model and a frontier model. Labeler bias is shared across candidates, so it does not change rankings, but labeler quality matters. We therefore validate the labeler's issue labels against legal annotators on a stratified sample.
Ground truth. Selection quality is measured against downstream performance on labeled post-cutoff target cases. For summarization, this is ROUGE-L and BERTScore plus expert ratings of legal correctness on a subset. For judgment prediction, it is macro-F1.
Two metrics summarize selection quality: Kendall's $\tau$ between each criterion's ranking and the downstream ranking, and regret. Regret is the downstream gap between the model a criterion selects and the best model in the pool.
Baselines
The naive reference point is to select the last checkpoint of the default recipe. That means plain fine-tuning from the source that is most lexically similar to the target, following Santosh et al.'s finding that similarity predicts transfer. We also report a random pick, so that regret numbers have a floor.
The standard label-free criterion is target perplexity on the public, pre-cutoff pool. This is the signal ProMALex optimizes. ProMALex reports average multilingual perplexity of 5.468, against 7.104 for a shared adapter and 8.338 for MoELoRA (ProMALex). We will test whether that ordering holds downstream.
The strongest label-free competitor is temporal-holdout perplexity: perplexity computed on the post-cutoff pool. It removes the contamination objection to perplexity outright. For this reason it, and not pre-cutoff perplexity, is the bar the proposal must clear.
We also run two other cheap coherence signals:
- word-level embedding coherence of cluster top terms, in the TC-W2V style of Greene and Cross;
- NPMI.
These isolate whether document-level labeling, the key ingredient borrowed from Stammbach et al., is doing the work. Stammbach et al. report that LLM word-level ratings correlate with humans at Spearman $\rho \approx 0.64$, against 0.43 for NPMI.
For judgment prediction, the downstream reference is JPO's reported Article F1 of 90.8 and Charge F1 of 89.7 on CAIL2018 (JPO). For summarization, it is Santosh et al.'s BVA-Ext→UK-Abs result, where silver augmentation moved ROUGE-L/BERTScore from 0.544/0.837 to 0.546/0.839. These anchor the absolute scale of our candidate pools. We found no reported result on Lacuna for the rank correlation or regret of any selection criterion in cross-jurisdiction legal transfer, so those comparisons are new.
Fair comparison. All criteria see the same unlabeled pools, the same candidate set, and the same compute envelope per candidate. Any criterion with a knob, such as the cluster grid or the perplexity context length, gets the same small tuning budget on a separate development jurisdiction.
What counts as winning. Coherence beats a baseline if it achieves higher $\tau$ and lower regret than temporal-holdout perplexity on post-cutoff downstream performance. The margin must exceed the spread across bootstrap resamples of the case pool, and it must hold in both testbeds.
Ablations. We isolate each part of the method by swapping document labels for top-word coherence, swapping the representation clustering for random partitions, and computing coherence on pre-cutoff versus post-cutoff pools.
Why not just…?
Why not just compute perplexity on post-cutoff text? This is the real threat. It removes leakage cheaply, and if it ranks candidates as well as coherence does, our contribution reduces to "use fresh text."
Our bet is based on the perplexity paradox from topic modeling, together with the Legal-Pegasus erasure result. Adversarial adaptation can leave fluency in the target register intact while destroying the legal distinctions that drive downstream quality. Perplexity sees the first and misses the second.
The head-to-head $\tau$ and regret comparison settles this, as does the specific check of whether each criterion ranks gradient-reversal Legal-Pegasus below its plainly fine-tuned version. If temporal perplexity ties coherence, we will report that as the finding.
Why not just label twenty target cases and select by ROUGE or F1? A small labeled development set is the obvious practical alternative. For summarization it may well win. We treat it as a reference curve rather than something to argue away.
We measure how many labeled target cases a supervised selector needs to match coherence-based selection. The coherence signal is valuable only if that break-even number is large relative to the cost of expert legal annotation. Where the only affordable labels come from public, likely leaked cases, the supervised selector is also exposed to contamination. We test that exposure directly: the same selector is scored with pre-cutoff labels and with post-cutoff labels.
Why not just pick the source jurisdiction by dataset similarity? Santosh et al. found that lexical and semantic similarity predicts transfer. ProMALex's dataset embedding similarity formalizes that finding. But similarity is a property of datasets, not of models. It cannot rank adaptation methods or checkpoints trained from the same source, which is where most of our candidate pool lives. We include it as the naive source-choice prior and report how much regret remains after using it.
Why not just have an LLM judge rate each candidate's outputs? Rating generated summaries directly is plausible. It needs a task-specific rubric, and it inherits the judge's own possible exposure to public judgments, a risk Stammbach et al. left unexamined. It also cannot score encoder or adapter models that produce no text.
We run it as an additional baseline on the summarization testbed, with the same labeler model, so any difference comes from rating outputs versus measuring organization, not from judge quality.
Proposed Experiments
Correlation study. In each testbed, we compute every criterion on both pre-cutoff and post-cutoff unlabeled pools and correlate it with post-cutoff downstream performance. We report $\tau$, top-1 regret and top-5 regret, with bootstrap intervals over cases.
Inflation index. Using CAIL2018 against the JPO-Dataset, and pre-cutoff against post-cutoff pools elsewhere, we measure how much each criterion's candidate ranking changes when the selection text moves from public to fresh. A contamination-robust criterion should give nearly the same ranking on both, which would let practitioners use it even on public data.
Planted contamination. This is the decisive check. We copy a subset of candidates and continue training the copies on the target's pre-cutoff selection pool, in three forms: verbatim, paraphrased, and machine-translated into another language, mirroring the attack of Yao et al.. The copies are thus contaminated on exactly the text used for selection. Their post-cutoff downstream performance should not improve, and we verify this.
We then ask, for each criterion, how often a contaminated copy is promoted above its clean original. Perplexity on the pool should be fooled, at least by the verbatim form. The question is whether doctrinal coherence is also fooled. Memorization could make representations cluster more tidily, and if it does, the contamination-robustness half of the claim fails. We will report that.
Erasure detection. On the Santosh et al. grid, we test whether coherence, unlike perplexity, reproduces the known drop from gradient-reversal adaptation on Legal-Pegasus. We also sweep the strength of the gradient-reversal penalty and check whether coherence falls monotonically as the penalty grows.
Kill criteria. The proposal fails on selection if coherence does not beat temporal-holdout perplexity on $\tau$ in at least one testbed with intervals that exclude a tie. It fails on robustness if planted contamination promotes contaminated copies under coherence at a rate comparable to perplexity. It fails on the method's central premise if the labeler's issue labels disagree with legal annotators so often that purity reflects the labeler's own inconsistency rather than differences between models. We will measure this by comparing labeler–annotator agreement against the spread of purity scores across candidates.
References
- Revisiting Automated Topic Model Evaluation with Large Language Models
- The Sensitivity of Topic Coherence Evaluation to Topic Cardinality
- Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization
- ProMALex: Progressive Modular Adapters for Multi-Jurisdictional Legal Language Modeling
- Data Contamination Can Cross Language Barriers
- Studying and improving reasoning in humans and machines
- LeKUBE: A Knowledge Update BEnchmark for Legal Domain
- Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark
- JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction
- Evaluation of Thematic Coherence in Microblogs
- Unveiling the Political Agenda of the European Parliament Plenary: A Topical Analysis