Large language models (LLMs) and retrieval systems face a fundamental conflict between computational efficiency and semantic coherence. Because standard Transformer architectures scale quadratically with sequence length, long documents are typically divided into smaller, manageable "chunks" or "passages." However, as Jyun-Yu Jiang et al. observe in their work on QDS-Transformer, this strategy creates contextual fragmentation: a loss of the overarching structure and semantic flow that defines a coherent document.
The Mechanics of Information Loss
Fragmentation occurs at multiple granularities, each degrading the model's ability to reason. At the most basic level, arbitrary segmentation—often based on simple token counts—causes "hard truncation." Lin Li et al. note in Long Legal Article Question Answering via Cascaded Key Segment Learning that this process frequently severs sentences or logical units mid-thought.
This physical splitting leads to three primary types of context loss:
- Structural Disconnection: Documents are rarely flat sequences. They rely on hierarchies like headers and sub-headers. Hui Wan et al. argue in Does Structure Matter? Encoding Documents for Machine Reading Comprehension that sliding window approaches separate content from its "ancestor chain," leaving the model unable to determine the specific scope or conditions governing a piece of information.
- Semantic Decoupling: When training or processing, segmenting text can decouple entities from the facts that follow them. Nayeon Lee et al. found that this fragmentation weakens a model's ability to associate topics with relevant details, leading to identity and attribution errors in Factuality Enhanced Language Models for Open-Ended Text Generation.
- Dependency Severing: In technical domains like telecommunications or law, interdependent details are often spread across sections. Omar Erak et al. highlight that fixed-size blocks sever these logical connections, preventing systems from providing a coherent context for complex standards in Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards.
The Retrieval Dilemma
Contextual fragmentation is particularly acute in Retrieval-Augmented Generation (RAG). Most RAG pipelines retrieve isolated snippets based on semantic similarity. As the authors of ChatClimate explain, this prevents the model from synthesizing information that spans different chapters, as the selected chunks lack the broader document's perspective.
Even if a retriever finds the "correct" chunk, that chunk may be useless if the necessary context—such as a pronoun reference or a chart description—resides in a different segment. Jaewoo Lee et al. demonstrate in Unified Multi-Modal Interleaved Document Representation for Information Retrieval that without a holistic representation, systems fail to understand how a specific paragraph relates to distant visual elements or structural markers.
Strategies for Restoration
To solve this, researchers are moving toward "context-aware" representations that bridge the gap between independent segments.
Late and Landmark Embeddings Instead of chunking text before embedding it, Michael Günther et al. propose Late Chunking. By applying the embedding model to the entire document first and only then pooling the representations into chunks, the model captures relationships between distant parts of the text. Similarly, Kun Luo et al. introduced Landmark Embedding, which uses special tokens to mark boundaries within a coherent context, allowing for chunking-free retrieval that maintains semantic continuity.
Hierarchical and Graph Representations Another approach involves modeling the document's global structure explicitly. Irene Li et al. developed HiPool, which uses Graph Neural Networks to model relationships between chunks, allowing for information "leakage" or overlap that restores the continuous flow. Qian Tang et al. extend this in GraphMPA by using community-based summarization to help models form a global understanding of a corpus.
Cross-Segment Attention Finally, some methods attempt to restore the attention mechanism itself. Jingbo Yang et al. developed KVLink, which uses "link tokens" to allow tokens to attend to information across segment boundaries in the KV cache, effectively stitching together the fragmented global context during inference.
By moving away from isolated, arbitrary chunks toward structure-aware and contextually-linked representations, these methods allow models to move beyond local pattern matching toward true document-level reasoning.