Language models do well on recall questions in professional domains and poorly on the tasks professionals actually need. On AECBench, a building-and-construction benchmark ordered from rote memorization up to writing compliant design documents, models score above 95% at the bottom level, and no model passes 60/100 on the top "creation" level (AECBench). The usual reading of this drop is that models cannot reason or compose. This note proposes a different explanation and a way to test it. In real deployments, a long building code, insurance policy, or technical manual does not fit usefully into the model's input. A retrieval step therefore keeps the passages that look relevant and discards the rest. The passages it discards tend to be the governing exceptions, numeric thresholds, and cross-referenced clauses that decide whether an answer complies. The model then writes a fluent, confident answer that is faithful to what it was shown and wrong under the full standard.
We call this an omission hallucination: an answer that is consistent with the retrieved context but violates a constraint the retrieval left out. Current hallucination checkers grade answers against the retrieved context, so they cannot see this error. They should mark these answers clean. The project builds a level-linked diagnostic that assigns each high-level failure to a specific lower-level cause: the constraint was never in the window, it was in the window but the model did not find it, it was found but misapplied, or the model invented it. The project then tests whether a simple rule that keeps constraint-bearing clauses and follows their cross-references closes the gap at a fixed token budget.
Core Claim
The claim has three parts, and each can be falsified. First, a substantial share of top-level failures on domain tasks are caused by context management rather than by reasoning. The test is to insert the missing clause back into the model's input and see whether the answer becomes correct. Second, existing faithfulness detectors, which check answers against retrieved evidence, systematically miss these failures. Third, a retention policy aware of constraints recovers much of the lost accuracy without increasing the token budget, and it does so by keeping the right clauses, not by keeping more text.
If inserting the missing clause recovers few failures, the cliff really is a reasoning problem. In that case we report this negative result and drop the third claim.
How This Relates to Prior Work
Several papers motivate the gap. The hierarchical domain benchmarks document the drop from easy to hard levels but do not explain it. AECBench uses a five-level ladder based on Bloom's taxonomy and judges answers with an LLM; it does not control what context the model sees. INS-S1 (An Industrial-Scale Insurance LLM) trains on hierarchical, verifiable insurance data and reports 0.6% hallucination on HHEM. Its questions are curated and self-contained, so they say nothing about failures caused by retrieval. ToolBeHonest (ToolBeHonest) is the closest structural precedent. It links three diagnostic levels (detecting whether a task is solvable, planning a solution, analyzing which tool is missing) to hallucination, and it finds that models score 90+ on solvable tasks but collapse on unsolvable ones. Its environments are small and synthetic, and the "missing" element there is a tool, not a clause dropped by retrieval. We borrow its idea of levels that diagnose one another. We differ in making the absence of information an experimental variable that we control.
Other work constrains the design. On long inputs, RepoDebug (RepoDebug) shows accuracy falling as inputs grow, with bug identification dropping from 51.48% to 43.07%. The relational-model study (Context Window Failures in Relational Foundation Models) shows raw high-volume inputs giving $R^2 \le 0.18$, against 0.65 after manual aggregation. Both results argue against simply passing the whole document. On the other side, CODEFILTER (Impact-driven Context Filtering) shows that pruning over 80% of retrieved tokens can raise accuracy. DyCoder (Partial Dependency Graph retrieval; Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generation) shows that removing multi-hop expansion costs 6 to 9%. Together these mark the tension we study: filtering helps on average, but it may cut exactly the distant dependencies that matter.
Two papers supply methods we reuse. RT4CHART (Retromorphic Testing with Hierarchical Verification) breaks an answer into atomic claims and verifies them first locally, against single chunks, then globally, against the full retrieved context. We reuse this decomposition, but we verify against the full standard. Its own stated limitation is that it judges faithfulness only to retrieved context, and our omission class is defined by that blind spot. DyCoder's select-visit-expand loop over a call graph supplies the expansion procedure. Building codes and policies contain explicit cross-references ("subject to clause 5.3.2"). These play the same role for a regulation that function calls play for a codebase.
Research Program
Building the testbed. We start from AECBench items at the Reasoning, Calculation and Application levels. For each item, domain annotators mark the gold set of governing clauses in the source standard, including exceptions and every clause reachable by cross-reference. AECBench was curated from official building codes, so we use those standards as the source.
Paired probes at lower levels. Each top-level item gets two companion items. A level-1 probe asks the model to quote the clause that governs a given condition. A level-2 probe asks how two gold clauses interact, for example which one takes precedence. Every probe is run in the same context the top-level item saw.
Controlled context conditions. We vary what the model receives in six ways: closed-book; the full standard in a long context window; BM25 top-$k$; dense top-$k$; CODEFILTER-style impact filtering adapted to text; and an oracle that contains only the gold clauses.
Attribution of each wrong answer. For every wrong top-level answer, the procedure works as follows:
- If a gold clause the answer violates was absent from the context, we label the failure window-omission. We confirm the label by checking whether inserting that clause flips the answer to correct.
- If the clause was present but the paired level-1 probe fails, we label it in-window miss.
- If the level-1 probe succeeds but the level-2 probe fails, or the clause is applied wrongly, we label it misapplication.
- If the answer cites a constraint that exists nowhere in the standard, we label it fabrication.
Retention policy. The method we test is deliberately simple. Before applying the budget cut, we mark clauses that carry rules: mandatory modal verbs such as "shall" or "shall not", numeric thresholds, and exception phrases. We then expand along explicit cross-references from any retrieved clause, following DyCoder's multi-hop procedure. Rule-bearing clauses get priority within the same token budget as the baselines.
Second domain. To check that the effect is not specific to building codes, we repeat the attribution on insurance policy items. This depends on access to INSEva-style items with source policies. If such access proves unavailable, we will say so rather than substitute synthetic text.
Baselines
Closed-book. The model answers with no retrieved context. This is the naive reference point. It tells us how much any context helps and how much of the cliff comes from parametric knowledge alone.
Full long context. The entire standard is placed in the input, truncated only at the model's maximum length. This is the reviewer's default competitor. RepoDebug and the Animus study suggest it degrades, but neither tested regulatory text. If full context matches the oracle condition, our omission story fails for capable long-context models, and we will report that.
BM25 and dense top-$k$ retrieval. Each is run at token budgets of 2k, 8k and 32k. These represent standard retrieval-augmented generation practice.
Impact filtering, the strongest existing context method. We train a CODEFILTER-style classifier that labels each chunk as helping, neutral or hurting, using our training split of AECBench items. This baseline matters because it learns which chunks help, so it could in principle learn to keep constraints, which would make our handcrafted retention rule unnecessary. Its paper reports an average 3% exact-match gain over plain retrieval, over 10% on instances that contain harmful chunks, and over 80% token reduction, all on code completion. No reported result exists on Lacuna for it on regulatory text.
Matched-budget ablation. This isolates the key ingredient of our policy. We replace rule-bearing and cross-referenced clauses with the same number of randomly chosen extra clauses, and separately with extra top-$k$ similarity chunks, at identical tokens. If our policy does not beat both, the gain comes from volume rather than from constraint content.
Fairness conditions. All conditions share the same models, prompts, budgets and decoding settings. Impact filtering gets the same training split and the same tuning budget we spend on the retention thresholds.
Detection baselines. We use RT4CHART with a GPT-4o mini judge, which its paper costs at $0.0104 per sample; the file gives no exact F1 numbers. We also use the decompose-and-aggregate long-context detector (Towards Long Context Hallucination Detection), which reports 67.22% balanced accuracy against 57.42% for GPT-4o on BookSum. Its authors note it does not transfer across domains zero-shot, so we fine-tune it on our training split.
What counts as beating a baseline. For context methods, the retention policy must give higher top-level compliance accuracy than impact filtering at equal tokens, and that gain must be concentrated in window-omission failures. For detection, the claim requires that both baseline detectors show markedly lower recall on omission failures than on fabrications. Re-running RT4CHART against the full standard instead of the retrieved chunks should recover most of that recall.
Why not just…?
Why not just put the whole standard in a long context window? This is the most serious threat, because current context windows can hold an entire code. The evidence against it comes from other settings: RepoDebug's accuracy drop with length, and Animus's $R^2 \le 0.18$ on raw inputs. Neither measures regulatory documents. Our full-context condition settles the question directly. If it matches the oracle on top-level items, retrieval is not the bottleneck for frontier models. The finding then narrows to budget-limited deployments, and we will say so.
Why not just use a better learned filter or reranker? Impact filtering is exactly this alternative, and it is included as a baseline with an equal training budget. If it matches our rule-based retention, the honest conclusion is that learned filtering already captures constraints. The contribution would then reduce to the diagnostic and the detector finding. We would accept that outcome.
Why not just prompt the model to reason step by step? INS-S1 reports that free-form step-by-step reasoning adds noise on exact regulatory lookups and that a rigid reasoning format worked better for actuarial math. AECBench reports that reasoning models still stay below 60 on creation tasks. More importantly, step-by-step reasoning cannot recover a clause that is not in the input. Our attribution measures this directly: we run every context condition with and without such prompting and check whether the reasoning prompts reduce window-omission failures or only misapplication failures.
Why not just run an existing hallucination detector? Detectors such as RT4CHART grade answers against what was retrieved. An omission failure is, by construction, faithful to what was retrieved. Our detection experiment measures how often this blind spot occurs. If the baseline detectors already flag omission failures at high recall, the second claim is false.
Decisive Checks
The central experiment is the oracle-insertion test. For each wrong top-level answer under top-$k$ and filtered contexts, we add only the missing gold clauses and re-query the model. We will consider claim one supported if insertion recovers at least a quarter of failures across the evaluated models. We will consider it refuted if insertion recovers well below that and the paired level-1 and level-2 probes show the failures are mostly misapplication.
The second check plots the share of window-omission failures against filtering aggressiveness. The prediction is that this share rises as the budget shrinks, even where average accuracy stays flat or improves. That pattern would be direct evidence for the tension between filtering and multi-hop dependencies.
The third check reports detector recall split by failure class. The prediction is a sharp recall gap between omission failures and fabrications for detectors graded on retrieved context, and a gap that closes when the same detectors are graded on the full standard.
The final check is the matched-budget ablation of the retention policy, reported per failure class. If the policy helps on top-level compliance only through the window-omission class, that confirms the mechanism. If it helps uniformly across all classes, or not at all, the mechanism is not confirmed.
References
- AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field
- An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
- Context Window Failures in Relational Foundation Models
- Impact-driven Context Filtering For Cross-file Code Completion
- Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generation
- Retromorphic Testing with Hierarchical Verification for Hallucination Detection in RAG
- Towards Long Context Hallucination Detection