Standard language models treat text as a flat, one-dimensional sequence of tokens. While self-attention mechanisms theoretically allow every token to interact with every other token, the structural distance between a concept introduced in a document's preface and a detail buried in its conclusion grows linearly with text length. In long documents, this linear layout creates a representation bottleneck: critical logical connections—such as an entity mentioned on page 1 and its coreferent pronoun on page 10—are diluted by thousands of intervening tokens.
To overcome this limitation, a prominent research direction models documents as multi-level heterogeneous graphs. Instead of a flat sequence, text is structured as a network where nodes represent different semantic units (such as words, entity mentions, sentences, paragraphs, sections, or external knowledge concepts) and edges represent the linguistic, structural, or logical relationships between them. By transforming text from a sequence into a topology, these models fundamentally alter how information propagates across a document.
Shortcutting Long-Range Dependencies
Why do graph representations drastically improve performance on multi-sentence reasoning and relation extraction? The core intuition lies in the concept of a small-world network. In a standard sequence, connecting information across distant sentences requires propagating signals through many layers of sequential processing. A graph representation bypasses this by introducing structural "shortcuts."
As shown by Fenia Christopoulou et al. in Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network, incorporating non-local edges—such as coreference links and sentence-level adjacencies—reduces the path length between distant mentions to a fraction of their sequential distance.
To exploit these shortcuts, architectures often introduce intermediate structural hubs. For example, Difeng Wang et al. in Global-to-Local Neural Networks for Document-Level Relation Extraction and Fenia Christopoulou et al. in Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs utilized heterogeneous graphs containing mention, entity, and sentence nodes. In these networks, sentence nodes act as routers. If two entities are mentioned in entirely different parts of a document, the model does not need to traverse the text sequentially; it passes information from the first entity node to its parent sentence node, which then routes it directly to other sentences or entities. This localized routing allows relational graph convolutions to capture global context efficiently without suffering from the vanishing gradients or attention dilution common in long sequences.
Navigating Multi-Scale Granularity
Documents are inherently hierarchical: words form sentences, sentences form paragraphs, and paragraphs form sections. Flat sequence models force the network to learn these boundaries implicitly. Heterogeneous graphs make them explicit, allowing models to compute representations at multiple scales simultaneously.
In Heterogeneous Graph Neural Networks for Extractive Document Summarization, Danqing Wang et al. designed a graph with separate node types for words and sentences. This architecture allows information to flow bi-directionally across different levels of granularity: word nodes aggregate local semantic meaning and pass it up to sentence nodes, while sentence nodes provide document-wide context back down to guide word-level importance. The authors demonstrated that this multi-level architecture naturally scales from single-document to multi-document tasks by simply introducing higher-level document nodes that link shared word nodes across different source files.
A similar multi-layered approach was leveraged by Pradeep Muthukrishnan et al. in Simultaneous Similarity Learning and Feature-Weight Learning for Document Clustering, where a dual-layered graph of documents and keywords created a mutual feedback loop: refining keyword weights improves document similarity metrics, which in turn refines the keyword weights.
Preserving Logical and Visual Layouts
For structured or visually rich documents—such as legal codes, technical specifications, or web pages—the physical or sequential proximity of text can be highly misleading. A section header might be physically adjacent to a paragraph, but logically connected to a sub-clause pages away. Heterogeneous graphs preserve the native logical organization of the document regardless of its linear layout.
To capture this, Jan Buchmann et al. in Document Structure in Long Document Transformers represented documents as ordered graphs where nodes correspond to structural elements like sections and paragraphs, encoding explicit tree-like relationships. For web documents, Qifan Wang in WebFormer: The Web-page Transformer for Structure Information Extraction introduced graph attention mechanisms that restrict communication to immediate parent, child, and sibling nodes within the HTML DOM tree, forcing the model to respect the document's native hierarchy rather than treating it as a flat string.
This structural alignment is particularly powerful for precise information retrieval and pruning. For example, Jiajie Jin et al. in Hierarchical Document Refinement for Long-context Retrieval-augmented Generation developed an XML-based document tree that allows retrieval-augmented generation (RAG) systems to prune entire irrelevant branches of a document hierarchy at once, drastically reducing token overhead while maintaining logical flow. In the legal domain, Vuong T. Pham et al. in Ontology-based Solution for Building an Intelligent Searching System on Traffic Law Documents mapped the formal hierarchy of parts, chapters, and clauses into an ontology graph, enabling precise penalty calculations by directly mirroring administrative structures.
Decoupling Structure from Reasoning
Why does this graph-based framing work so well computationally? It allows models to separate the extraction of linguistic structure from the actual logical reasoning.
The authors of Document-level Relation Extraction with Dual-tier Heterogeneous Graph, Zhenyu Zhang et al., implemented a dual-tier heterogeneous graph that instantiates this exact separation:
- The First Tier models syntactic and hierarchical relationships between words and sentences to capture local linguistic context.
- The Second Tier focuses on coreference and co-occurrence between entities to facilitate multi-hop inference across the entire document.
By isolating these concerns, the model avoids confounding local grammatical dependencies with document-wide logical relationships. Furthermore, Shuang Zeng et al. in Double Graph Based Reasoning for Document-level Relation Extraction demonstrated that a dual-stage graph architecture—which first captures interactions between specific mentions and then collapses those mentions into consolidated entity nodes—retains fine-grained contextual details while enabling clean, document-wide reasoning.