Recognizing Non-Linear Two-Dimensional Structures

87 concepts69 papers1 proposal

Most sequence-to-sequence models excel at processing linear text because human language is fundamentally a one-dimensional sequence of tokens read from left to right. However, many critical domains—such as mathematics, chemistry, music notation, and user interfaces—defy this linear assumption. A mathematical formula or a chemical diagram is a two-dimensional puzzle where the spatial arrangement of symbols dictates their semantic meaning.

As noted by Anssi Kanervisto et al. in Image-to-Markup Generation with Coarse-to-Fine Attention, traditional optical character recognition (OCR) systems fail in these domains because they are bound to a linear processing assumption. To accurately parse structured documents, models must move beyond simple sequence mapping and learn to interpret non-linear spatial layouts.

Why Linear Models Fail in Two Dimensions

Why can't we simply flatten a two-dimensional image into a one-dimensional string using standard sequence-to-sequence architectures? The breakdown happens due to two core challenges: spatial ambiguity and hierarchical nesting.

In standard text, a character's context is almost entirely defined by its horizontal neighbors. In mathematical or chemical notation, a symbol's meaning changes completely based on its vertical or diagonal alignment. Stephen M. Watt demonstrates in Determining Points on Handwritten Mathematical Symbols that distinguishing between a subscript, a superscript, or standard multiplication requires precise knowledge of character baselines and typographic metrics. Without these explicit reference points, the spatial arrangement remains ambiguous, leading to incorrect semantic interpretations.

Linear Text:       x  +  y  =  z      (Purely horizontal)

2D Notation:       x² + 1
                   ------             (Vertical alignment & 
                     y                hierarchical nesting)

Furthermore, structures like fractions, matrices, and nested radicals introduce a deep hierarchy. A fraction contains a numerator and a denominator separated by a vinculum, both of which can contain their own nested fractions or exponents. Pegah Ahadian et al. highlight this in MNIST-Fraction: Enhancing Math Education with AI-Driven Fraction Detection and Analysis, showing that vertical alignment is just as critical as horizontal sequence. When standard sequence decoders attempt to process these layouts, they lack the multi-dimensional logic required to track multi-line alignments, resulting in severe structural errors and alignment drift (Felix M. Schmitt-Koopmann et al. in MathNet: A Data-Centric Approach for Printed Mathematical Expression Recognition).

Mapping Visual Space to Structural Trees and Graphs

To bridge the gap between a 2D visual input and a 1D markup string (such as LaTeX), modern architectures explicitly model the underlying structural hierarchy rather than treating it as a flat sequence.

One prominent strategy replaces linear sequence generation with tree- or forest-structured decoding. In PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer, Tongkun Guan et al. treat the target markup as a "position forest." By explicitly modeling the 2D hierarchical structure of symbols, the model builds a structural map that guides it through complex spatial relationships, preventing alignment drift in deeply nested expressions. Similarly, Hanbo Cheng et al. in Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition utilize bidirectional training over tree structures to capture multi-dimensional dependencies that standard OCR misses.

Alternatively, the layout can be parsed as a graph. In Visual Parsing with Query-Driven Global Graph Attention (QD-GGA): Preliminary Results for Handwritten Math Formula Recognition, the authors treat formulas as graph-structured data rather than linear sequences, mapping spatial relationships directly to graph edges. This graph-based paradigm extends to other non-linear domains, such as logographic air-writing, where Kaiqi Huang et al. use graph attention networks to capture complex intersecting strokes and geometric relationships rather than a simple temporal sequence of movements (Finger in Camera Speaks Everything: Unconstrained Air-Writing for Real-World).

Spatial-Aware Attention and Coverage Tracking

How do these models maintain focus across non-linear layouts without skipping or repeating symbols? Researchers have introduced three core mechanisms to enforce spatial discipline within attention frameworks:

  • Multi-Scale and Coarse-to-Fine Attention: To recognize both large structural elements (like a fraction line or a matrix bracket) and tiny details (like a subscript or a decimal point), models must operate at multiple granularities. Jianshu Zhang et al. demonstrate in Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition that combining dense encoders with multi-scale attention successfully captures fine-grained details alongside high-level structural context, resolving the "disappearing symbol" problem.
  • Coverage Modeling: Vanilla Transformers are permutation-invariant and lack an inherent memory of where they have looked in an image. To fix this, Wenqi Zhao et al. developed CoMER: Modeling Coverage for Transformer-Based Handwritten Mathematical Expression Recognition. By tracking which parts of a 2D formula have already been processed, the model significantly reduces omissions and repetitions in long, complex expressions.
  • Explicit Spatial Supervision: Rather than forcing the network to implicitly learn layout rules from data, some architectures explicitly supervise spatial awareness. In SSAN: A Symbol Spatial-Aware Network for Handwritten Mathematical Expression Recognition, Haoran Zhang et al. introduce an auxiliary task that forces the network to predict the 2D spatial distribution of symbols. This explicitly trains the model's "sense of space," directly mitigating errors caused by non-sequential structural hierarchies like nested radicals.

A Universal Paradigm for Structured Layouts

While mathematical expression recognition serves as the primary testbed for these techniques, the core principles of two-dimensional structural recognition apply universally across any domain where layout carries semantic weight.

In chemical notation, Peng Tang et al. demonstrate in Chemical Symbol Feature Set for Handwritten Chemical Symbol Recognition that handwritten chemical formulas require integrating alphanumeric characters, bonds, and structural rings into non-linear spatial arrangements. Similarly, sheet music involves multiple staves, overlapping voices, and intricate symbolic relationships. In LEGATO: Large-scale End-to-end Generalizable Approach to Typeset OMR, Guang Yang shows that transcribing full-page music requires capturing dense spatial dependencies entirely absent from sequential text.

Even in modern digital tasks like front-end code generation, models confront these exact spatial constraints. As highlighted by Zhiyu Lin et al. in WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code, multimodal models frequently struggle with precise element sizing and complex horizontal arrangements. This underscores that mastering two-dimensional structural logic remains a critical frontier across computer vision and structured data parsing.

Key Foundational Work

Visual Grammars for Visual Languages (1987) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 67 papers
  2. Version 1 · 2026-10-04 · 81 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_two_dimensional_structural_recognition_bfdd9ef8,
  title = {two-dimensional structural recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/two-dimensional-structural-recognition/rdr_bfdd9ef85d3a4436910da29da3de8295},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_two_dimensional_structural_recognition_bfdd9ef8_v2,
  title = {two-dimensional structural recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/two-dimensional-structural-recognition/rdr_bfdd9ef85d3a4436910da29da3de8295/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake