Language models reading long contracts are reported to fail when a clause and the definition or exception that governs it sit far apart. ContractScrub documents a sharp recall drop once a reference is more than 10,000 characters from its target. No model it tested exceeds 0.650 F1, and the best reaches only 0.750 macro recall (GPT-5.5). The usual explanation is the context window: attention weakens over long spans. There is a second explanation that nobody has separated from the first. Distant exceptions ("Notwithstanding Section 2.1, the Tenant shall not…") tend to share few words and little meaning with the obligation they override. Any system that first retrieves relevant chunks, whether by dense embeddings (texts mapped to vectors so that similar meanings land close together) or by keyword matching, will rank the decisive clause low. A third confound makes both explanations hard to test. The public contracts behind these benchmarks (CUAD, ContractNLI) have been online for years and are likely in pretraining data, so models may partly recall answers rather than reason about them.
This proposal builds a procedural generator of full-length contracts in which the logical dependency structure, the character distance between dependent clauses, the similarity of the operative clause to the query, and the novelty of the text are set independently. With it, one can ask which factor actually drives failure on long legal documents. The same generator yields fresh, never-published test instances, which gives a contamination-free benchmark as a side effect. The generator is the tool; the causal decomposition of the "distance penalty" is the question it answers.
Core Claim
We hypothesize the following. For retrieval-based pipelines, failures on cross-clause exceptions are explained mainly by the rank of the operative clause under the retriever, not by its token distance. For full-context long-window models, distance matters but interacts strongly with how similar the exception looks to the queried rule. Part of the headline accuracy on static contract benchmarks reflects familiarity with published text; it will shrink when the same logical structure is realized with invented party names, defined terms, and section numbering.
The contribution is a factorial design that makes these claims testable. Existing work can only observe them correlationally. ContractScrub measures distance on real contracts, where distance, similarity, and familiarity move together. DyVal generates reasoning problems from directed acyclic graphs (DAGs) to avoid leakage, but only in formal domains such as arithmetic and graph reachability. We carry DyVal's idea of generating from a controlled graph into legal natural language and add a control axis DyVal lacks: how retrievable each node is.
How the Generator Works
Each instance starts from a dependency graph over clauses. Nodes are defined terms, general obligations, permissions, and exceptions. Edges encode "defines," "is excepted by," "is conditioned on," and "cross-references." The gold answer to a question is computed from the graph, not from the text. An example question: "Is the Licensee permitted to sublicense to an Affiliate after termination?" Answers use ContractNLI-style labels (entailed, contradicted, not mentioned) together with the supporting clause set.
Clauses are realized through attorney-written templates drawn from real drafting patterns. For modality we use the agent-specific deontic distinctions catalogued in LEXDEMOD. That work shows "shall" makes up 44.6% of modal triggers and flips between obligation and entitlement depending on the party, so we realize obligations with both "shall" and non-modal triggers such as "represents and warrants." The generated clauses are embedded in filler sections sampled from real CUAD-style boilerplate. This gives total length and layout realistic scale while the operative logic stays synthetic.
Four factors are set independently. The first is logical depth, meaning the number of hops from the question to the deciding clause. The second is character distance between dependent clauses, varied by moving filler, with the 10,000-character threshold from ContractScrub as a reference point. The third is surface and semantic similarity between the query and the deciding clause. It is controlled by choosing template variants with high or low word overlap and measured after generation as the clause's rank under a fixed panel of retrievers. The fourth is novelty. At one extreme are verbatim CUAD phrasings with familiar party names. At the other are invented defined terms, reordered section numbering, and paraphrased boilerplate.
Two kinds of exceptions appear. Explicit exceptions name their target ("except as provided in Section 7.3(b)"). Implicit exceptions override by scope without naming it ("Notwithstanding anything herein to the contrary, no party shall…"). This distinction matters for the reviewer objections below.
Label validity is the main risk. Labels are fixed by construction, and paraphrasing is restricted to meaning-preserving template variants. Practicing attorneys will label a stratified sample of generated instances blind. We will report their agreement with the generator's labels and discard template families that fall below a set threshold.
Research Program
The first stage builds the generator and validates it. Validation has two parts: attorney agreement with the gold labels, and correspondence with reality. For correspondence, we check that model rankings on low-novelty generated contracts correlate with rankings on the original ContractNLI and ContractScrub. If they do not, the generator measures something else and the program stops to diagnose why.
The second stage runs the factorial study across three system families: full-context long-window models, retrieval-augmented pipelines, and structured-memory systems. For each family we fit a model of accuracy as a function of depth, distance, retriever rank of the deciding clause, novelty, and exception type. The quantity of interest is how much each factor explains, and in particular whether distance still matters once retriever rank is held fixed.
The third stage estimates contamination. We build isomorphic twins of existing ContractNLI items: the same logical graph re-realized with fresh surface text. The accuracy gap between an original item and its twin estimates how much the published text is being recalled. We cross-check it against per-token surprise on the original text, following the memorization evidence in Yax et al..
The fourth stage tests one simple remedy suggested by the decomposition. When a chunk is retrieved, a deterministic cross-reference and defined-term resolver adds every clause it names, and every clause that uses a defined term it relies on. This is our key ingredient, and its ablation is the sharpest test of the claim. It should close much of the gap for explicit exceptions. It should do nothing for implicit ones, whose failures would then be attributable to how retrievable the clause is.
Baselines
The naive reference is a long-window model given the whole contract with no retrieval. It runs at the same maximum context across all distance conditions with a fixed prompt, with and without chain-of-thought. ContractScrub reports chain-of-thought adds roughly +0.10 recall for top models, so both variants are needed. Its best reported figure is 0.750 macro recall (GPT-5.5) on real contracts. The two questions are whether that model shows a distance penalty on generated contracts when similarity is held fixed, and how its accuracy moves from original ContractNLI items to their fresh twins.
The strongest retrieval competitor is hybrid dense-then-lexical reranking in the style of LeSeR. LeSeR fine-tunes a BGE dense retriever and then reranks its top 20 candidates with BM25. On ObliQA it reports mAP@10 of 0.6655, against 0.6237 for BM25 alone and 0.1077 for the dense retriever alone. It is the most relevant competitor because it was built specifically for the gap between legal meaning and embedding similarity. We run it with the same chunking, the same reader model, and the same top-k budget as every other pipeline, and with the retriever fine-tuned on an equal amount of generated training data. A method beats it if it improves exception-level accuracy in the low-similarity, implicit-exception cells without losing accuracy elsewhere.
The strongest structured-memory competitors are PAKTON and JERR. PAKTON organizes a contract as a hierarchical tree and runs BM25, dense, and graph retrieval with iterative questioning; it reports Recall@1 of 53.14% on ContractNLI. JERR builds a DAG from chunk summaries and searches it with Monte Carlo tree search; it reports 86.39% on QuALITY, but has no reported legal result. Both are run with the same reader model and a matched per-query token budget. Token cost is reported because JERR's own comparison with ReadAgent (44.33k versus 79.98k tokens) shows cost varies widely. A compression baseline, CORE-RAG, which learns to compress retrieved text with task reward, is included to test whether aggressive compression drops exception clauses. It reports +3.3 exact match on Natural Questions at about 3% of tokens; no reported result found on Lacuna for CORE-RAG on legal contracts. The key ablation is the cross-reference resolver switched on and off inside the LeSeR pipeline, with everything else held fixed.
Why not just…?
Why not just perturb existing CUAD and ContractNLI contracts instead of generating new ones? Renaming parties and shuffling sections is cheaper and more realistic, and it is a real threat to the generator's necessity. It addresses novelty but cannot vary similarity independently of distance, because the text of each exception stays fixed. We include perturbed-static sets as a condition. If they reproduce every finding of the full factorial design, including the rank-versus-distance decomposition, the generator adds nothing beyond contamination control, and we will report that.
Why not just put the whole contract in a long-context model? This is the naive baseline, and it may work well at the lengths in question. ContractScrub's contracts are only 10 to 15 pages, and frontier models still cap at 0.750 recall there, which suggests window size alone is not the fix. The factorial design will show whether full-context failures follow distance, similarity, or depth. DyVal's finding that graph depth hurts accuracy faster than width predicts that depth, not length, dominates.
Why not just measure contamination with perplexity? Low surprise on benchmark text, as in Yax et al., shows a model has seen the text. It does not show that familiarity inflates accuracy, or by how much. The isomorphic-twin gap measures that directly, and we report both so readers can see whether they agree.
Why not just add cross-reference resolution and stop? If explicit exceptions dominated real contracts, this would be most of the answer. ContractNLI's evidence of non-local "negation by exception" and its contradiction F1 of about 0.36, against about 0.85 for entailment (Koreeda and Manning), suggest many overrides are implicit. The resolver ablation, split by exception type, decides this empirically.
Decisive Checks
The central test is a within-system comparison on retrieval pipelines. Take cells with the same retriever rank for the deciding clause and vary character distance from under 2,000 to over 20,000 characters. If accuracy stays flat, the ContractScrub distance penalty is a retrievability effect for these systems. If it still falls, distance has its own cost. A second test is validity: model rankings on low-novelty generated contracts must correlate positively with rankings on ContractNLI and ContractScrub, and attorney agreement with generator labels must meet the preset threshold. A third is the contamination estimate, the accuracy drop from original ContractNLI items to their fresh twins, reported per model with confidence intervals. A fourth is the resolver ablation: it should recover accuracy on explicit exceptions and leave implicit-exception accuracy unchanged. If the resolver fixes implicit exceptions too, our account of what makes them hard is wrong. Metrics are exception-level accuracy, per-label F1 with contradiction reported separately, evidence recall, and tokens per query.
References
- ContractScrub: A benchmark for final review of legal contracts
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Agent-Specific Deontic Modality Detection in Legal Language
- Studying and improving reasoning in humans and machines
- 1-800-SHARED-TASKS at RegNLP: Lexical Reranking of Semantic Retrieval (LeSeR) for Regulatory Question Answering
- PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements
- Joint Enhancement of Relational Reasoning for Long-Context LLMs
- Less Is More: Elevating RAG via Performance-Driven Context Compression
- ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts