Search systems for legal precedent must decide which past cases matter most for a new one. "Matters" here comes in degrees. A case with the same charges and the same statutory articles is more useful than one that shares a charge but was sentenced under different provisions, and that case is in turn more useful than one that only describes a similar street fight. Current legal pre-training throws this gradation away. Pre-training here means training a text encoder on millions of unlabeled court documents before any task-specific tuning. The strongest such model teaches its encoder a yes/no lesson: this pair of cases is a match, every other case in the batch is not. We propose to replace that binary lesson with a graded one. The structured fields of rulings (charges, cited provisions, sentence length) are already used to manufacture training pairs. We will use them instead to build ordered relevance ladders, and pre-train the encoder with a pairwise ranking loss. That loss pushes each rung above the one below it by a margin that grows with the grade gap.
The opportunity is concrete because the evaluation already exists. LeCaRDv2 (LeCaRDv2) annotates relevance in graded form along characterization, penalty, and procedure. A binary-contrastive pre-training objective cannot be expected to produce the fine ordering those labels reward. If graded pre-training helps anywhere, it should help on graded metrics over partially relevant cases. If it does not, the result tells the field that binary contrast already captures what the structure offers.
Core Claim
Caseformer pre-trains on 5 million Chinese criminal case documents. It uses two contrastive tasks. One separates cases with similar facts but different judgments. The other matches cases with similar judgments by their facts, mining positives with a provision-weighted frequency score called LP-ICF. Both tasks are binary: a pseudo-positive against negatives. As a result, a case that shares one of two charges with the anchor is treated exactly like an unrelated case. This is the false-negative problem that the pairwise-ranking literature identifies as the main failure mode of hard-negative mining. Our claim is that this binarization costs ranking quality. A graded margin objective over the same structural signal should improve NDCG on graded relevance and reduce errors on partially relevant cases. It should do this without hurting recall, because it keeps intermediate cases close to the anchor instead of pushing them away.
The ranking side supplies the method. Pairwise hinge losses with localized contrastive estimation separated near-miss passages in the CIP TREC 2022 re-ranker, but only during downstream fine-tuning. Adaptive margins drove accuracy in graph-attention recommendation. We move these ideas upstream into pre-training. PEER also puts a multi-level ranking loss into pre-training: it ranks original tokens above plausible replacements above nonsense, and reports a GLUE average of 86.77 against ELECTRA-Base's 86.42. PEER differs from our work in two ways. Its ranks come from a generator's token probabilities on general text, and they operate on tokens rather than documents. Our grades come from domain structure and apply to whole cases. LinkBERT models cross-document relations as a three-way classification. SAPBERT uses pairwise metric learning with online hard-pair mining, but only for binary synonymy. Both motivate the gap: relational pre-training so far either classifies or contrasts, and neither orders.
Research Program
The first component is a grading function over pairs of case documents, computed only from structured fields. For an anchor case and a candidate, the grade rises with three things: the overlap of their charge sets, the overlap of their cited provisions (weighted by LP-ICF so that common boilerplate articles count little), and the closeness of their sentence-length buckets. The grade function has four levels. Grade 3 means the same charges, the same provisions, and an adjacent penalty bucket. Grade 2 means the same charges with differing provisions or penalty. Grade 1 means partial charge overlap or the same chapter of the criminal code. Grade 0 means the remaining cases. The thresholds will be fixed on a small held-out sample of LeCaRDv2 training annotations and then frozen. We will report how well the grades agree with expert labels, measured by Kendall's tau, before any model is trained. This is a cheap sanity check: if the grades do not correlate with expert relevance, the project stops here.
The second component is the objective. Following Caseformer, the anchor side is the factual description alone, since that is what a query looks like at test time. The candidate side is the full case. For each anchor we assemble a list of candidates spanning the grade levels, mined with BM25 and LP-ICF so that the low grades contain lexically similar cases rather than easy random ones. We then minimize a graded pairwise hinge loss:
$$ \mathcal{L}_{\text{rank}} = \sum_{(i,j):\, g_i > g_j} \max\big(0,\; m\,(g_i - g_j) - (s(q,d_i) - s(q,d_j))\big) $$
Here $q$ is the anchor's fact description and $d_i$, $d_j$ are candidate cases. Their grades are $g_i$ and $g_j$. The function $s$ is the encoder's relevance score: an inner product for the dual encoder, a scalar head for the cross-encoder. The constant $m$ is a base margin, so a larger grade gap demands a larger score gap. This loss is added to masked language modeling on the same corpus, as Caseformer does. Pair counts grow quadratically in list length. We will therefore cap lists at 16 candidates per anchor, four per grade, and sample pairs within a list. Under this cap the cost per step stays close to a contrastive batch of equal size.
The third component is analysis. We will measure where the gains land, broken down by expert relevance grade and by LeCaRDv2 dimension. LeCaRDv2 reports procedure as the hardest dimension. Our grades encode characterization and penalty but not procedure, so we predict gains on the first two and little change on procedure. A uniform gain across all dimensions would suggest the improvement comes from something other than the grade structure.
Baselines
The naive reference points are BM25 and a generic pre-trained encoder. On LeCaRDv2 zero-shot, BM25 reaches Recall@100 of 0.6262, against 0.1165 for Chinese-BERT-WWM (LeCaRDv2). Any pre-trained model that loses to BM25 zero-shot has failed. We will run BM25 with the same tokenization and candidate pool used for every neural model.
The strongest existing method is Caseformer. On LeCaRD it reports zero-shot R@500 of 0.8323 against BERT's 0.3351, and fine-tuned R@100 of 0.6111 against SAILER's 0.5937 (Caseformer). We found no reported result on Lacuna for Caseformer on LeCaRDv2, and none for NDCG on LeCaRDv2 from any model. The fine-tuned reference on LeCaRDv2 is RetroMAE, with Recall@1000 of 0.8174. For a fair comparison we will reproduce Caseformer's recipe, using its released checkpoint if available and otherwise reimplementing it. We will use the identical backbone, corpus, tokenizer, step count, and batch size, changing only the objective. Each method gets the same hyperparameter search budget, eight configurations each. Beating Caseformer means three things. First, a statistically significant NDCG@30 gain on LeCaRDv2 overall relevance, in both zero-shot and fine-tuned settings, averaged over three pre-training seeds. Second, recall within one point of Caseformer's. Third, a reproduced LeCaRD R@100 at or above Caseformer's own reported 0.6111.
Two ablations isolate the key ingredient. In the binarized ablation, the identical mined lists are collapsed so that grade 3 is positive and everything else negative; this is Caseformer's supervision on our pairs. In the shuffled-grade ablation, grades are randomly permuted within each list while their distribution is kept. If the shuffled model matches the graded one, the ordering carries no information. If the binarized model matches it, the gain comes from our mining rather than from the grading.
Why not just…?
Why not just use Caseformer's contrastive loss with soft labels weighted by the grade? This is the most serious threat. InfoNCE (the standard contrastive loss) with targets proportional to the grade also refuses to treat partial matches as pure negatives, and it may capture most of the benefit. We will not argue around it. We will run it as a full baseline, using the same lists and the same budget. If soft-label InfoNCE matches the hinge loss, the contribution narrows to "graded supervision from structure," and that becomes the paper's claim. The pairwise form matters only if it wins on fine ordering, specifically NDCG@10 among cases with expert grade 2 or higher.
Why not just fine-tune a general or Caseformer encoder with a pairwise loss on LeCaRDv2's graded labels? Supervised graded fine-tuning is cheap, and the CIP results show pairwise fine-tuning is effective. However, LeCaRDv2 has only 800 queries, while structured pseudo-grades cover millions of cases. Evidence that pre-training dominates the fine-tuning recipe (PEFT comparison for code) suggests the gap will not close downstream. Our fine-tuned comparison tests this directly. Caseformer and our model both receive the same graded pairwise fine-tuning. If they then converge, graded pre-training is redundant.
Why not just predict the query's charges and provisions, then rank candidates by structural overlap? Candidate cases carry structured fields, so a charge classifier followed by LP-ICF overlap is a strong non-neural-ranking pipeline. It may well match expert characterization labels, which are themselves charge-driven. We will build this baseline with a classifier fine-tuned from the same backbone. We expect it to fall short in two places: penalty and fact-level nuance, and cases whose charge prediction is wrong. If it matches our model, the encoder has learned nothing beyond the labels used to construct its grades.
Why not just scale a general model? Prior domain-specific results argue against this. LinkBERT's 340M biomedical model beat GPT-3 on MMLU professional medicine (50.7% vs 38.7%), and the generic BERT zero-shot numbers on both legal datasets are very low. We therefore treat this alternative as settled for encoders of this size and do not spend compute on it.
Proposed Experiments
The main study pre-trains three dual encoders and three cross-encoders with identical budgets: Caseformer-style, binarized-on-our-lists, and graded. It also trains the soft-label and shuffled variants for the dual encoder only, to limit cost. All are evaluated on LeCaRDv2 and LeCaRD, zero-shot and after fine-tuning. Metrics are NDCG@10 and NDCG@30 on graded labels, Recall@100 and Recall@1000, and MAP, reported per relevance dimension. Pre-training corpora will exclude every LeCaRD and LeCaRDv2 query and candidate document, checked by near-duplicate hashing, because structured pseudo-labels could otherwise leak the annotation criterion.
The decisive check has three parts. The graded model must beat both the binarized and the shuffled ablations on NDCG over partially relevant cases. Its advantage must concentrate in characterization and penalty rather than procedure. And among cases experts mark relevant, it should rank fewer below the cutoff at which the Caseformer-style model rejects them. That last measure is our direct estimate of the false-negative damage that binary pre-training causes. A secondary, smaller probe applies the same grading idea to COLIEE 2020, using cited-statute overlap. We found no reported numbers on Lacuna for this dataset, so it serves only as a test of whether the gain survives outside Chinese criminal rulings.
References
- LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset
- Caseformer: Pre-training for Legal Case Retrieval
- CIP at TREC 2022 Deep Learning Track
- Graph Attention Collaborative Similarity Embedding for Recommender System
- PEER: Pre-training ELECTRA Extended by Ranking
- LinkBERT: Pretraining Language Models with Document Links
- Self-alignment Pre-training for Biomedical Entity Representations
- A Comparative Study of PEFT Methods for Python Code Generation