Overcoming the Document Independence Assumption

128 concepts120 papers5 proposals

When modern language models or information retrieval systems process a large corpus, they typically treat it as a flat collection of isolated text chunks. This design choice is known as the document independence assumption. Under this assumption, whether a system is pretraining on web pages, retrieving evidence for a question, or summarizing scientific literature, it evaluates each document or passage as an independent island of information.

While treating documents as independent simplifies computation, it creates a severe relational context gap. Human knowledge is inherently networked; ideas do not exist in a vacuum. They are explicitly linked via hyperlinks, academic citations, legal precedents, and cross-entity relationships. When models ignore this topology, they lose the semantic bridges necessary to navigate and reason over complex information landscapes.

Why Flat Representations Fail Complex Reasoning

To understand why the document independence assumption breaks down, consider a multi-hop question analyzed by Asai et al. in their work on retrieving reasoning paths: "When was the football club founded in which Walter Otto Davis played?"

A traditional retriever operating under the document independence assumption will score documents based on their lexical or vector similarity to the query. It will successfully find a passage about Walter Otto Davis stating that he played for Millwall F.C. However, the founding date of Millwall F.C. resides in an entirely different document. Because the original query does not contain the word "Millwall," a standard independent retriever will assign the second document a low relevance score, missing it entirely.

[ Query: Walter Otto Davis club foundation? ]
                     │
                     ▼ (Independent Retrieval)
┌────────────────────────────────────────┐   ┌────────────────────────────────────────┐
│ Document A: Walter Otto Davis played   │   │ Document B: Millwall F.C. was founded  │
│ for Millwall F.C.                      │   │ in 1885.                               │
└────────────────────────────────────────┘   └────────────────────────────────────────┘
        SCORE: HIGH (Matches Query)                  SCORE: LOW (No Query Overlap)

As Cheng et al. demonstrate with the CORE framework, treating evidence pieces as independent units causes an immediate loss of relational context. Without maintaining explicit links between a primary source and its secondary references, models fail to reconstruct the multi-hop reasoning chains required to answer complex questions. The system cannot follow the thread because it cannot see the connections between the pages.

Injecting Topology into Pretraining

One way to overcome this limitation is to dismantle the document independence assumption during the model's initial training phase. Instead of feeding a language model a randomized sequence of text segments, we can structure the pretraining corpus as a graph of interconnected documents.

This is the core intuition behind LinkBERT, introduced by Leskovec et al.. Instead of focusing solely on local coherence within a single document, LinkBERT places linked documents—such as pages connected by hyperlinks or citations—into the same context window during pretraining.

Standard Pretraining:  [ Document A (Page 1) ] ──> [ Document A (Page 2) ]
Link-Aware Pretraining: [ Document A (Source) ] ──> [ Document B (Cited/Linked Target) ]

By forcing the model to predict text transitions across document boundaries, the architecture internalizes the semantic bridges that humans use to navigate knowledge. The model learns not just what a document says, but how its concepts contextualize and expand upon the concepts in a neighboring document.

Graph-Aware Retrieval and Generation

In Retrieval-Augmented Generation (RAG), moving past the document independence assumption means shifting from retrieving isolated text chunks to retrieving structured subgraphs.

In the scientific domain, knowledge is distributed across a dense web of citations where the true impact or nuance of a paper is contingent on its relationship to preceding and succeeding work. To capture this, Lei et al. developed CG-RAG, a framework that entangles lexical and semantic signals directly within a citation graph. Rather than treating papers as flat text blocks, CG-RAG preserves the relational neighborhood, allowing the language model to understand the broader academic conversation surrounding a claim.

This topological approach is equally powerful when grounding models in structured factual data. Wen et al. introduced MindMap, a framework that prompts large language models to construct an internal "mind map" by extracting and traversing the explicit structural relationships within a Knowledge Graph (KG). By anchoring the model's reasoning in the topological connections between entities, MindMap prevents the hallucinations common in standard RAG systems that rely on disjointed text snippets.

Domain-Specific Interconnectedness

The necessity of abandoning the independent document view becomes starkest in highly specialized fields like legal research. Legal systems are explicitly relational; a single case opinion is meaningful only when viewed through its web of citations, statutory references, and binding precedents.

To address this, Qiu et al. developed CaseLink, an inductive graph learning framework designed for legal case retrieval. Traditional legal search engines treat cases as isolated text entities, which frequently fails because two cases might use completely different vocabulary to describe the same underlying legal doctrine. By modeling the intrinsic graph structure of citations and shared legal charges, CaseLink leverages latent semantic connections that text matching alone misses.

By transitioning from a flat, bag-of-documents paradigm to a graph-centric architecture, these diverse frameworks ensure that models preserve the connective tissue of human knowledge, leading to more robust reasoning, fewer hallucinations, and a deeper contextual understanding.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_document_independence_assumption_29be1465,
  title = {document independence assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-independence-assumption/rdr_29be14658c20409d9015b6de8596a95f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_document_independence_assumption_29be1465_v1,
  title = {document independence assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-independence-assumption/rdr_29be14658c20409d9015b6de8596a95f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake