Reading Order Detection for Non-linear Document Layouts

This direction is split into other directions: reading order ambiguity. Its last version stays readable and citable here.

Version 1 of 178 papers93 conceptsfrozen 2026-10-04

Traditional Optical Character Recognition (OCR) treats text as a one-dimensional stream, much like speech. This "text-as-speech" paradigm assumes that if you scan a page from top-to-bottom and left-to-right, you will capture the logical flow of information. However, real-world documents—ranging from multi-column historical newspapers and complex ESG reports to the overlapping panels of manga—rarely follow such a simple path.

The research direction of Reading Order Detection (ROD) seeks to move beyond rigid spatial heuristics. Instead of assuming a linear sequence, these models treat document understanding as a 2D structural puzzle, learning to reconstruct the "human read-order" from visual and spatial cues.

The Failure of Linear Heuristics

Standard OCR engines often "choke" on complex layouts because they rely on fixed sorting rules. As Wenzhen Zhu and colleagues observe in DocBed, digitizing historical newspapers is like solving a jigsaw puzzle; simple geometric heuristics frequently merge unrelated columns or lose the narrative flow when articles jump across visual breaks or advertisements.

This problem is not limited to old paper. Modern "slide-like" documents, such as ESG annual reports, use decorative typography and non-standard layouts that break the assumption of a continuous flow. Lin Gui notes in A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports that relying on font size or proximity often leads to incorrect hierarchical nesting because the logical reading path is non-linear.

From Sequences to Graphs and Trees

To solve this, researchers are shifting from modeling reading order as a permutation (a single list) to modeling it as a structure.

Chong Zhang argues in Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding that a single ordered list is fundamentally flawed for complex documents. Instead, they propose representing reading order as a Directed Acyclic Graph (DAG). This allows the model to capture branching paths—such as when a reader might choose to read a sidebar before or after the main text—more accurately reflecting human cognition.

Similarly, Yue Zhang and the team at Fudan University introduced PDF-to-Tree, which parses disorganized text blocks into a hierarchical tree of sections and subsections. By moving away from linear sequences, these models prevent "context fragmentation," where sidebars or footnotes interrupt the main narrative.

Learning the "Human" Order via Pre-training

How does a model learn what "order" even means without a human telling it? The breakthrough has come from large-scale layout-aware pre-training.

In LayoutReader, Zilong Wang and collaborators created ReadingBank, a dataset of 500,000 documents where the "ground truth" reading order was extracted from the underlying metadata of Word documents. This allowed a seq2seq model to learn the relationship between 2D coordinates and logical sequence.

Other models, like ERNIE-Layout by Hao Tian, treat layout as a "third modality" alongside text and vision. By introducing a "Reading Order Prediction" task during training, the model learns to rearrange shuffled text blocks into their correct sequence, making it robust to the multi-column layouts that baffle traditional models.

Domain-Specific Challenges: Manga and Manuscripts

The most extreme tests for these ideas are found in graphic storytelling and historical artifacts.

In manga, text bubbles are scattered across panels that may overlap or follow diagonal boundaries. Ragav Sachdeva and the Oxford Visual Geometry Group developed Magi, which treats the entire page as a graph. By processing the page "in-context," the model can determine "who said what" by associating text bubbles with characters based on their spatial and narrative relationship, rather than just their vertical position.

Historical manuscripts present a different challenge: vertical text flows and irregular structural inserts. Yongxin Shi notes in M5HisDoc that standard tools fail here because they rely on modern structural priors. These documents require models that can interpret symbols within their original, often damaged, non-linear spatial contexts.

The Shift to 2D Awareness

The common thread across this research is the rejection of the 1D scanning line. Whether it is Junyeop Lee’s SATRN, which uses 2D self-attention to recognize curved or multi-line text, or Dongsheng Wang’s DocLLM, which uses "disentangled" attention to handle spatial coordinates, the field is moving toward a purely spatial understanding of documents. By treating a page as a 2D map rather than a 1D tape, these systems are finally beginning to "read" with the same flexibility as a human.

Papers in this version

Showing 60 of 78 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reading_order_detection_for_non_linear_d_1f546cd3,
  title = {Reading Order Detection for Non-linear Document Layouts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reading-order-detection-for-non-linear-document-layouts/rdr_1f546cd3e02941d89b7de5682dc3e7aa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reading_order_detection_for_non_linear_d_1f546cd3_v1,
  title = {Reading Order Detection for Non-linear Document Layouts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reading-order-detection-for-non-linear-document-layouts/rdr_1f546cd3e02941d89b7de5682dc3e7aa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}