Hierarchical Neural Architectures for Structured Document Understanding

This direction is merged into another direction: hierarchical document structure. Its last version stays readable and citable here.

Version 1 of 1161 papers170 conceptsfrozen 2026-10-04

Standard language models view text as a "flat" sequence—a linear string of tokens where the 1st word is related to the 10,000th only through the brute force of self-attention. However, human-authored documents are rarely flat. They are hierarchical, organized into a nested structure of words, sentences, paragraphs, and sections.

Research in hierarchical document modeling seeks to move beyond the sequence-to-sequence paradigm by injecting this structural inductive bias directly into neural architectures. By treating a document as a tree or a graph rather than a list, these models gain the ability to navigate long-form content, distinguish between core arguments and supporting details, and maintain coherence across massive contexts.

The Problem of Structural Blindness

When a model like BERT or GPT processes a scientific paper, it often treats the "Introduction" and the "Acknowledgments" with equal priority. This "structural blindness" leads to several failure modes:

  1. The Head Distribution Problem: Models often over-attend to the beginning of a document because they lack the "anchors" needed to navigate to the middle or end SAPGraph.
  2. Context Dilution: In a flat sequence, the semantic signal of a section header is lost as it is averaged out across thousands of tokens TravelBERT.
  3. Redundancy: Without understanding the logical relationship between segments (e.g., a summary vs. a detailed explanation), models struggle to extract unique "facts" Fact-level Extractive Summarization.

Hierarchical Attention and Multi-Granular Encoding

The most direct solution is to mirror the document's physical hierarchy in the model's attention mechanism. Instead of every token attending to every other token, researchers like Tobias Rohde and Yinhan Liu proposed the Hierarchical Attention Transformer (HAT).

HAT introduces specialized sentence-level attention layers. The model first encodes tokens into sentence representations, and then "zooms out" to allow these sentence representations to attend to one another. This multi-level reasoning allows the model to maintain a high-level "map" of the document while still having access to fine-grained token details Hierarchical Learning for Generation with Long Source Sequences. Similarly, Zilong Wang and colleagues developed MGDoc, which extends this to document images, modeling the nesting of words within paragraphs and paragraphs within pages MGDoc.

Documents as Graphs and Trees

Beyond simple nesting, documents possess a logical "rhetorical" structure. A sentence might support a claim, contradict a previous finding, or summarize a section.

Models like DISCOBERT, developed by Jiacheng Xu and Zhe Gan, use Graph Convolutional Networks to model Rhetorical Structure Theory (RST). By treating sub-sentence units as nodes in a graph, the model can "leapfrog" over irrelevant text to find semantically related content that is spatially distant Discourse-Aware Neural Extractive Model for Text Summarization.

For machine reading comprehension, Hui Wan proposed TreeJC, which encodes a paragraph's "ancestor chain"—the sequence of headers and sub-headers leading to it. This provides the model with a "tree slice" of context, ensuring that a paragraph about "Climate" in a section on "Mars" isn't confused with one about "Earth" Does Structure Matter? Encoding Documents for Machine Reading Comprehension.

Anchoring with Meta-Information

Another powerful intuition is treating document structure as a guide for attention. In TG-Net, Wang Chen treats the title not as part of the text, but as a "query" that guides the encoding of the body. This forces the model to prioritize information that aligns with the high-level summary provided by the author Title-Guided Encoding for Keyphrase Generation.

Similarly, HIBRIDS by Shuyang Cao and Lu Wang injects learnable "hierarchical biases" into the attention matrix. These biases act like a magnetic pull, encouraging tokens within the same section to attend more strongly to each other than to tokens in unrelated sections HIBRIDS.

Why Structure Works

The success of these models stems from a core realization: structure is a proxy for salience. Human writers use headers, bullet points, and discourse markers to signal what is important. By explicitly modeling these signals, hierarchical architectures achieve several things:

Ultimately, hierarchical modeling transforms the task of "reading" from a linear scan into a structured traversal of an information map.

Papers in this version

Showing 60 of 161 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_neural_architectures_for_st_6884ff52,
  title = {Hierarchical Neural Architectures for Structured Document Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-neural-architectures-for-structured-document-understanding/rdr_6884ff524afb4ed5ad6ad4b0b3daa5da},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_neural_architectures_for_st_6884ff52_v1,
  title = {Hierarchical Neural Architectures for Structured Document Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-neural-architectures-for-structured-document-understanding/rdr_6884ff524afb4ed5ad6ad4b0b3daa5da/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}