Catching Memorized Legal Verdicts Across Jurisdictions

When a language model scores well on a legal benchmark, we usually cannot tell whether it reasoned from the facts and the law or simply remembered the case. This matters more in law than in most domains. The public Chinese criminal datasets are large and widely scraped, so they are likely to leak into pretraining data. A model that has memorized them can also look competent while being "jurisdiction-locked," meaning it has learned one country's answers rather than how to apply a rule.

We propose a two-part method. The first part is a test that settles the question with ground truth. We take a codified civil-law case and rewrite the governing rule inside the prompt, for example by moving the theft-amount threshold for a sentencing bracket or redefining who counts as the victim. We then check whether the model's verdict follows the new rule or sticks to the original answer. The second part asks whether a cheap geometric fingerprint of the model's own reasoning text can predict that stickiness from the original case alone. If it can, the fingerprint works where rule-swapping is impractical, such as common-law tasks built on precedent.

The fingerprint comes from topological data analysis. This is a family of methods that summarizes the "shape" of a point cloud: how many separate clusters it forms and whether it contains loops, measured across distance scales. The full version, persistent homology, is too expensive to run on thousands of token states per answer. We therefore use a cheaper summary, the Euler characteristic curve, computed over many small overlapping windows of the reasoning trace.

Core Claim

Memorized and reasoned answers to the same legal question leave different shapes in the model's hidden states while it writes its reasoning. Three things are needed to make that claim useful:

  1. Labels. Codified law lets us generate ground-truth "memorized vs. reasoned" labels cheaply, through rule-swap stickiness and deliberately planted contamination.
  2. Cost. The shape signal must survive the cost constraint, so a windowed Euler characteristic curve must retain what full persistent homology captures.
  3. Transfer. A detector built on the topological signal should transfer from Chinese civil-law data to English common-law data better than detectors built on token probabilities, whose thresholds depend on vocabulary and language.

The novelty is not "TDA for contamination" in the abstract. It is the label source, the cost-bounded invariant, and the cross-jurisdiction transfer test together.

Where This Sits

Work that motivates the gap. - Benchmark Data Contamination of Large Language Models: A Survey organizes contamination into four levels, from semantic to label-level. It documents that surface rewording does not remove contamination: EvoEval rewrites of HumanEval cost 51 models an average of 39.4% of their score. - DyVal builds evaluation items on the fly from random directed acyclic graphs, meaning dependency graphs with no cycles. Models strong on static tests, phi-1.5 and WizardMath-13B, scored nearly zero on DyVal. GPT-4 produced "unsubstantiated responses," answers with no intermediate steps, in 33.8% of its failures. - DyVal's generator covers only arithmetic, logic and graph reachability, and it scores only final answers. We bring its generate-don't-store idea to legal rules and also inspect the trace.

Work that supplies the topological lens, and how we differ. - Understanding Chain-of-Thought in Large Language Models via Topological Data Analysis embeds reasoning steps with a sentence encoder and runs persistent homology. It finds an "Explore-then-Collapse" pattern: early branching is topologically complex and the final solution path collapses to something simple. - Persistent Topological Features in Large Language Models tracks layer-to-layer hidden-state topology with zigzag persistence on nearest-neighbor graphs. It warns that such signals are sensitive to the neighborhood parameter. - Neither paper studies memorization or law. Both use full persistent homology, whose worst-case cubic cost in the number of simplices is the bottleneck surveyed in A roadmap for the computation of persistent homology. - Our working hypothesis follows from their finding. A memorized verdict should look like a collapse with no exploration: a trace that is short in topological terms even when it is long in tokens.

Work that supplies the cost-saving machinery. - GPU Computation of the Euler Characteristic Curve for Imaging Data skips matrix reduction entirely. It counts cells on GPUs at over 5×10⁹ voxels per second, but only on image grids. - From Geometry to Topology: Inverse Theorems for Distributed Persistence shows that invariants computed on many small overlapping subsets can still pin down global geometry. It was tested only on circles and tori. - We combine the two on irregular point clouds of token states. The known cost of doing so is that Euler curves discard which loops exist and how long they live. Our experiments measure whether that loss matters here.

Work that shapes the legal side. - Legal Judgment Prediction via Topological Learning and JPO treat Chinese criminal judgment as a dependency chain from articles to charges to penalty. Despite its title, the first paper is topological only in that dependency-graph sense and computes no invariants. These dependency chains are what make our rule rewrites computable. - Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction shows that numeric amounts drive sentencing brackets. Legal Analysis and Victim Identification of Credit Card Fraud Cases shows that victim definitions differ across systems. Together they give us two realistic axes to perturb. - CitaLaw argues that statute-oriented verification does not carry over to precedent citation. This is exactly why rule-swapping cannot simply be repeated on common-law benchmarks, and why a transferable passive signal is worth having.

Research Program

The rule-swap generator. We start from CAIL2018 cases with deterministic sentencing logic, in particular property crimes whose bracket depends on an amount. We extract the factual elements: amount, role, victim, and prior offenses. The fine-grained elements of LEEC give an annotation scheme for this step. Each case's rule is encoded as a small program over those elements.

A swapped item keeps the facts verbatim. It states, inside the prompt, a fictional jurisdiction's version of the rule: a shifted threshold, a reassigned victim, or a reordered aggravating factor. The correct answer comes from running the program, much as DyVal derives answers from its generated graph.

Stickiness is defined over swapped items whose correct answer differs from the original label. It is the fraction of those items on which the model returns the original CAIL label. Each original case becomes an item-level label for that model:

  • Sticky: high stickiness, meaning recall is beating the stated rule.
  • Responsive: the model follows the new rule.

Planted contamination. Rule-swap labels are behavioral and could reflect poor instruction following rather than memory. We therefore also create contamination whose cause we know.

We fine-tune copies of an open-weights model on chosen subsets of test items. Two contamination levels from the survey's taxonomy are used: label-level (question plus answer) and data-level (question only). Each is applied at several repetition counts. Qwen2.5-7B is the natural choice because it is the backbone of the strongest CAIL system in the file.

The same planting is done on English items from LexGLUE's European Court of Human Rights task and from LegalBench rule-application tasks. The English items supply the transfer targets. Items from JPO's 2024–2026 case collection that postdate the base model's stated training cutoff serve as a natural clean reference.

The fingerprint. The point cloud for each answer is the set of hidden states of the generated reasoning tokens at a few fixed layers, compared by cosine distance. The trace is split into overlapping windows, for example 64 tokens with stride 32.

In each window we build the standard nested family of graphs that adds an edge whenever two points are within distance $\varepsilon$, together with the triangles those edges form. We record the Euler characteristic curve

$$\chi_w(\varepsilon) = V_w(\varepsilon) - E_w(\varepsilon) + T_w(\varepsilon),$$

where $V_w$, $E_w$ and $T_w$ count the points, edges and triangles of window $w$ at scale $\varepsilon$. Computing this requires only counting, with no matrix reduction, so it parallelizes on a GPU as in the imaging work. It also drops the per-loop bookkeeping that makes full persistent homology expensive.

The sequence of window curves, summarized as early-versus-late complexity and integrated area, feeds a small logistic-regression detector. The early-versus-late summary is our version of an exploration-then-collapse index. On a subsampled subset, full persistent homology for clusters and loops is computed as an expensive reference.

Baselines

Naive reference. The do-nothing reference point is the accuracy gap between pre-cutoff benchmark items and post-cutoff JPO items for the same model. It indicates that contamination exists at the population level but cannot flag individual items. Item-level detection at chance, an AUROC of 0.5, is the floor.

Token-probability membership inference. This is the matching-and-likelihood family described in the survey: score an item by how unsurprised the model is by its text. It is the cheapest and most widely used competitor.

Perturbation and calibration detectors. The strongest existing black-box methods in the file are CAP: Data Contamination Detection via Consistency Amplification and Data Contamination Calibration for Black-box LLMs. CAP measures how consistently a model answers under small rewrites. The calibration method corrects probability-based scores for item difficulty. They are strongest because they target the same item-level question and do not require access to the training corpus. No reported result was found on Lacuna for CAP or calibration-based detection on any legal benchmark, so we will establish their numbers ourselves. For task performance on CAIL2018, the reference is JPO's reported Article F1 of 90.8 and Charge F1 of 89.7 with Qwen2.5-7B (JPO). That figure tells us the base model is competent on CAIL, so stickiness cannot be blamed on incapacity.

Fairness of comparison. Every detector is scored on:

  • the same items;
  • the same generated traces where traces are used;
  • the same planted-contamination labels;
  • the same classifier family;
  • the same hyperparameter search budget, with equal trials per method.

We report GPU-hours per thousand items for each method.

Key ablation: shape without topology. This ablation isolates what topology adds. The same detector is given plain trace statistics: token length, number of reasoning steps, mean and variance of pairwise hidden-state distances, and the windowed distance spread. A second ablation replaces the windowed Euler curves with one global curve. A third replaces them with full persistent homology on subsampled points.

What counts as winning. We commit in advance to the following. Within Chinese data, the topological detector should match the best competitor. In the transfer settings, it should exceed both the best non-topological trace statistic and the strongest probability-based detector by at least 0.05 AUROC, with confidence intervals across planting seeds that do not overlap. If it beats CAP but not the plain trace statistics, the claim that topology is needed fails. We would then report that the cheap statistics suffice.

Why not just…?

Why not just check n-gram overlap against the training data? For deployed models the pretraining corpus is unknown, so overlap cannot be computed. EvoEval also shows that memorized structure survives rewording (survey). In our planted setting overlap with the fine-tuning set is perfect by construction. We use it only as an oracle sanity check, never as a competitor.

Why not just use token probabilities or perplexity? These are real competitors, and they are cheap. Our concern is transfer. Probability thresholds depend on vocabulary, script and phrasing, which is the motivation for the calibration work. A threshold tuned on Chinese criminal facts may misfire on English court opinions. The transfer experiment settles this directly. If calibrated probability scores transfer as well as the topological fingerprint, we will say so.

Why not just run the rule-swap test everywhere and skip topology? Where rules are codified, rule-swapping is the better detector, and we treat it as ground truth. It cannot be built cheaply for precedent-based items. Rewriting "the law" there means fabricating a consistent body of case law, which CitaLaw suggests is structurally different from editing a statute. Topology earns its place only if it carries the rule-swap signal into settings where the test cannot be built. That is what the civil-to-common-law transfer measures.

Why not just flag short or step-free answers, given DyVal's 33.8% unsubstantiated responses? This is the most serious threat. It is why trace statistics form the central ablation and are not a side baseline. Our specific bet concerns traces that are full length but recited. Planted label-level contamination with forced step-by-step prompting creates exactly these, so length and step count should fail on them. If they do not fail, the topological claim is dead.

Why not just evaluate on post-cutoff cases? Post-cutoff splits such as JPO's 2024–2026 cases detect contamination in aggregate. They cannot say which existing benchmark item is compromised, and they age as models are retrained. We use them as the naive reference, not as a substitute.

Decisive Checks

The program depends on four results, run in this order.

  1. Validity of the label source. Rule-swap stickiness must rise with planted contamination dose, from data-level to label-level and with more repetitions. It should stay low on post-cutoff items. If stickiness tracks instruction-following failures instead, the generator is not measuring memory and must be fixed before anything else.
  2. Signal presence. On Chinese items, sticky and planted-contaminated answers should show lower early-window complexity in the windowed Euler curves than responsive answers. This is the predicted "collapse without exploration." The effect must hold across several choices of window size and distance scale, given the sensitivity reported for neighborhood parameters (Persistent Topological Features in Large Language Models).
  3. Cost-fidelity. Windowed Euler curves should recover most of the detection AUROC of full persistent homology on the subsample while using a small fraction of its GPU-hours. We report the full cost-versus-AUROC curve rather than a single point.
  4. Transfer. A detector trained only on Chinese CAIL labels is evaluated on planted contamination in ECtHR and LegalBench. A held-out Chinese evaluation on LawBench separates the effect of language shift from the effect of legal-tradition shift. The topological detector must beat both the trace-statistics ablation and calibrated probability scores by the pre-committed margin.

A clear failure at step 4, combined with success at steps 1–3, would still leave a useful result: a contamination-resistant, rule-swap legal benchmark generator and an honest bound on what geometric fingerprints can show.