Where the Overlap Lives: Section-Level Diagnosis of Lexical Shortcuts in Document Similarity

Text embedding models often treat "these two texts share many words" as if it meant "these two texts mean the same thing." This is the lexical overlap bias. It is well documented for short sentence pairs. It has barely been examined for the long, structured documents where such models are now used to retrieve precedents, find related methods, or flag duplicates. Professional documents have a property that sentence benchmarks lack. They are built from functional sections, such as facts, claims, the court's reasoning, and the final judgment in a ruling, or background, method, and results in a paper. These sections differ sharply in how much they repeat across documents and in how much they decide what a document means. Two appellate rulings can share almost every sentence of their factual narrative and still reach opposite outcomes, a difference that sits in a few hundred tokens of reasoning and holding.

We propose to measure lexical overlap bias section by section rather than document-wide. The claim is that a document embedder's similarity score is driven mostly by word overlap in the long, repetitive sections, while the short sections that carry the outcome barely move it. Segmenting documents into functional sections is what makes this measurable. It lets us swap one section for another, stratify pairs by where their overlap falls, and correct similarity for overlap separately inside each section. We would build the diagnostic and the correction, test both on matched Chinese civil appeal pairs, and check whether the correction transfers to scientific and Indian legal documents.

Core Claim

Document-level overlap bias is largely a question of which sections the overlap comes from. Three things follow if that is right. First, an embedder's similarity on long documents should be predicted far better by overlap in high-volume sections (facts, headers, procedural boilerplate) than by agreement in outcome-bearing sections. Second, replacing a high-volume section with one from an unrelated document should move similarity much more than replacing the outcome section, even on a task where the outcome section is the only thing that matters. Third, removing the part of each section's similarity that its own lexical overlap predicts, and then combining sections with learned weights, should recover outcome sensitivity. It should do so better than either correcting for overlap at the whole-document level or reweighting raw section similarities.

Relation to Prior Work

Work on the bias itself motivates the proposal but stops at short text. PAWS built word-swap pairs that drop BERT from over 90% accuracy on Quora Question Pairs to 33–40%. SemAntoNeg found that the widely used all-MiniLM-L6-v2 scores 1.9% on "not [antonym]" paraphrases, against a 33.3% random baseline, while models tuned on natural language inference (deciding whether one sentence entails another) reach 93.6%. ALIGN-SIM showed that GPT-3-ada, top on downstream benchmarks at 90.23%, fails basic semantic distinction tests. Aiming beyond the Obvious split similarity datasets into "obvious" pairs, where word divergence agrees with the label, and "non-obvious" pairs, where it does not, using Jensen-Shannon divergence between word distributions. It found that obvious pairs make up more than half of every benchmark. We reuse that split, but compute it per section instead of per text pair.

Two methods supply tools that we use as baselines. AoE argues that cosine similarity saturates for high-overlap pairs and trains on angle differences instead. Neighboring Distribution Divergence masks the shared subsequence of two sentences and compares a masked language model's predictions at those positions. Both treat the input as one undifferentiated sequence. Neither can tell the difference between overlap in a boilerplate header and overlap in a holding.

On the decomposition side, two papers show that sections matter, though neither asks whether neural encoders are fooled within or across them. Neves et al. built pseudo-documents from selected rhetorical elements of biomedical abstracts and compared them with a string-similarity measure. Using only Challenge, Outcome, and Future Work raised F@10 from 0.45 for whole text to 0.90. That result is a direct ancestor of our approach. It also uses hand-picked sections, a single expert annotator, and no dense encoder. Shukla et al. evaluated legal summaries role by role. They found that ROUGE, a word-overlap metric, correlates with expert judgment at only 0.13–0.21. That supports our premise that overlap and legal validity come apart, but the paper does not examine the encoders.

AppealCase supplies our main data. It contains 10,000 matched first- and second-instance Chinese civil judgments, balanced 50/50 between affirmed and reversed. Each document is split by keyword rules into header, claim, facts, court's view, judgment, and footer. One point needs to be addressed directly. AppealCase reports that all 20 evaluated language models score below 50% F1 on predicting reversal from the first-instance document alone. That task involves a single document, so it cannot be explained by overlap between paired documents. We do not claim to explain that result. We use the matched pairs because they are a rare natural source of documents that overlap heavily yet disagree in outcome.

Research Program

The work has three parts.

The first is a contrast benchmark built from AppealCase. We pair second-instance rulings that share a cause of action and label each pair by whether the outcomes agree. Within each pair we compute word divergence separately for every section. We then stratify pairs so that fact-section overlap is matched between same-outcome and different-outcome pairs. This produces a document-level analogue of the obvious and non-obvious split, with the "obvious" cue located in a known section.

The second part is two diagnostics that require no training.

The first diagnostic is the overlap-reliance profile. We regress an encoder's document cosine on the per-section overlaps and the outcome label, and report the share of variance attributable to each section's overlap.

The second diagnostic is the section-swap sensitivity profile. For one section at a time, we replace it with the same section from a random same-cause document and record the drop in similarity. An embedder that behaves sensibly on outcome comparison should react most to swaps of the court's view and judgment. Our hypothesis is that general embedders react most to swaps of the facts.

The third part is the correction, called section-residualized similarity. For each section $k$ of a document pair, let $s_k$ be the encoder's cosine similarity between the two documents' $k$-th sections and $o_k$ their lexical overlap, measured as one minus Jensen-Shannon divergence. We fit a monotone function $f_k$ on unlabeled pairs, predicting $s_k$ from $o_k$. The pair score is

$$D = \sum_k w_k \,\big(s_k - f_k(o_k)\big),$$

where the weights $w_k$ are learned on a small labeled development split. Each term is the part of a section's similarity that its own word overlap does not explain. The encoder stays frozen, so the method can sit on top of any embedder, including AoE and NLI-tuned models.

Baselines

The naive reference point is whole-document cosine from a general sentence encoder such as all-MiniLM-L6-v2, run on documents chunked and mean-pooled. Next to it we report a pure word-overlap score, one minus Jensen-Shannon divergence over the whole document. Together these show how much of the encoder's behavior plain word counting already reproduces.

We also include a keyword oracle that reads the appellate disposition phrases in the judgment section. It exists to show that the outcome signal is lexically available. The research question is whether embedders use that signal when facts overlap, not whether the task can be solved at all. The oracle has no counterpart in the biomedical transfer setting.

The strongest existing encoder baselines are AoE and an NLI-tuned encoder of the kind that reached 93.6% on SemAntoNeg, both applied to whole documents. For long technical text, AoE reports 70.59 Spearman correlation on GitHub issue similarity against 60.38 for SimCSE (AoE). That is the strongest reported long-text result in the file. No reported result was found on Lacuna for any encoder on an outcome-agreement task built from AppealCase pairs. We therefore establish all baseline numbers ourselves.

On the biomedical transfer, the reference is Neves et al.'s hand-selected Challenge–Outcome–Future Work pseudo-documents at F@10 = 0.90, against 0.45 for whole text. Beating it means matching or exceeding 0.90 with learned rather than hand-picked section weights, using the same 562 documents and the same seven seeds.

Every method uses the same frozen encoder checkpoints, the same section segmentation, the same development split for any fitted parameters, and a matched hyperparameter search budget.

Two ablations isolate the key ingredient. Learned weights over raw section similarities, without the residual step, test whether the overlap correction adds anything beyond choosing sections. Whole-document residualization, which subtracts one global overlap fit with no sections, tests whether decomposition adds anything beyond overlap correction. The proposal beats the field only if it outperforms both ablations on the overlap-matched stratum of the contrast benchmark. The metric is AUC for outcome agreement, with bootstrap confidence intervals.

Why not just…?

Why not just embed only the judgment section? On AppealCase this is a real threat. Because the outcome is concentrated in one known section, selecting it may close most of the gap, just as section selection did for Neves et al.. The learned-weights-only ablation covers this baseline and includes hand selection as a special case. If it matches the residualized score on AppealCase, we report that honestly. Our claim then narrows to the diagnostic profiles plus the transfer settings. On Indian court rulings with rhetorical roles annotated by law experts (IN-Ext) and on the biomedical set, the relevant meaning is spread over several roles, so no single section can be selected in advance.

Why not just feed the whole document to a long-context model? Shukla et al. found that long-context abstractive models distort legal reasoning, and AppealCase reports that even large reasoning models stay below 50% F1 on its hardest task. Those findings concern generation and single-document prediction, not similarity, so they only suggest this baseline will struggle. To test it directly, we score pairs with a long-context cross-encoder on the overlap-matched stratum. A cross-encoder reads both documents jointly and outputs one score, so its comparison against cheap frozen embeddings with a correction is reported on accuracy and on compute.

Why not just use an NLI-tuned or angle-optimized encoder? Both fix short-sentence traps. We have no evidence that the fix survives mean pooling over thousands of tokens dominated by shared facts. Both encoders are run as baselines and also as the base encoder under our correction. If whole-document NLI or AoE already shows a flat overlap-reliance profile, the bias is not section-driven and our core claim is wrong.

Why not just subtract global word overlap? This is the whole-document residualization ablation. If it performs as well as the per-section version, the decomposition is unnecessary and the proposal reduces to a known style of debiasing.

Proposed Experiments

The first decisive check is whether the diagnostics show what the core claim predicts. On the AppealCase contrast benchmark, MiniLM, AoE, and NLI-tuned encoders should place most of their overlap-reliance variance on the facts and claim sections. Their section-swap profiles should be dominated by fact swaps rather than court's-view swaps. If the profiles turn out flat or already concentrated on the outcome sections, the core claim fails.

The second check is the overlap-matched stratum. Whole-document encoders should fall toward chance there while the keyword oracle stays high. The residualized score should recover a substantial share of the gap and beat both ablations.

The third check is transfer. We apply the AppealCase-fit residualization functions, refit only on unlabeled in-domain pairs, to the biomedical similarity set and to IN-Ext, using automatic rhetorical labelers. We compare against the 0.90 F@10 reference. Because those two datasets are small, with 562 and 50 documents, we treat transfer results as directional and report per-seed variance.

As a final sanity check, we score the section-level components on PAWS and SemAntoNeg. This confirms that the correction does not make the underlying encoder oversensitive to harmless rewording. Over-correction of that kind is a known failure of adversarial patches.