Standard language models process information like a conveyor belt—one token after another in a single linear sequence. This one-dimensional paradigm works beautifully for natural language text, where meaning flows sequentially from left to right. However, when applied to structured data like two-dimensional tables, this linear processing model breaks down.
To feed a table into a standard Transformer, it must first be "linearized"—flattened into a 1D string using formats like Markdown, CSV, or HTML. This serialization process introduces a fundamental architectural mismatch that degrades performance on tabular reasoning tasks.
The Linearization Bottleneck and Spatial Context Loss
When a 2D table is flattened into a 1D sequence, the inherent spatial relationships between cells are obscured. As noted by Hongzhi Zhang et al. in Table Fact Verification with Structure-Aware Transformer, linearization destroys the direct alignment between data cells and their corresponding headers or row neighbors.
Consider a cell located at row 10, column 5. In its native 2D grid, its meaning is instantly anchored by its vertical column header and its horizontal row context. Once linearized row-by-row, however, that cell is separated from its column header by hundreds of intervening tokens. Because standard positional embeddings only encode 1D sequential distance, the model must expend significant attention capacity simply to reconstruct the basic 2D grid topology. When the context window grows large, the model frequently fails to maintain these relationships, leading to misalignments and factual hallucinations.
Furthermore, flattening a table strips away what the authors of Open Domain Question Answering with A Unified Knowledge Interface call "spatial shortcuts." In a raw table, operations like sorting, ranking, or finding the top-ranked item rely on physical layout cues (such as row indices or column proximity). Converting these structures into fluent natural language narratives removes these physical shortcuts, forcing the model to perform complex logical reasoning over a long string rather than exploiting the visual and structural geometry of the grid.
Injecting Structure into Attention Mechanisms
To resolve this bottleneck without abandoning the Transformer architecture, researchers have modified the self-attention mechanism to make it natively aware of 2D geometry. Instead of allowing every token to attend to every other token uniformly, structure-aware models constrain or augment the attention matrix based on grid coordinates.
Standard Attention (1D Flattened):
[Header A] -> [Header B] -> [Cell A1] -> [Cell B1] -> [Cell A2] -> [Cell B2]
(Cell B2 is far from Header B in sequence distance)
Structure-Aware Attention (2D Masked):
[Header A] <---> [Cell A1] <---> [Cell A2] (Column Attention)
^ ^ ^
v v v
[Header B] <---> [Cell B1] <---> [Cell B2] (Row Attention)
In Table Fact Verification with Structure-Aware Transformer, the authors introduce custom attention masks that encode the table's 2D grid structure directly into the self-attention layer. This ensures that a cell token can immediately attend to its corresponding row and column headers regardless of how far apart they are in the serialized string.
A parallel approach is developed by Graham Neubig et al. in TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data, which uses a "vertical attention" mechanism. Instead of flattening the entire table into one long sequence, TaBERT processes individual rows in parallel alongside the column headers. The model then computes attention vertically across rows for the same column, allowing it to capture multi-dimensional dependencies and preserve the explicit links between headers and cell values.
Modeling Tables as Graphs and Hypergraphs
Linear text sequences inherently imply a strict ordering. However, relational tables possess a property called permutation invariance: swapping the order of two rows or two columns changes the serialized text sequence completely, but it does not alter the underlying meaning of the data. To capture this property natively, a prominent research direction treats tables as graphs rather than text streams.
As demonstrated by Xingyao Zhang et al. in A Graph Representation of Semi-structured Data for Web Question Answering, representing tables as graphs—where cells are nodes and structural alignments (rows and columns) are edges—prevents the loss of vertical context. This graph formulation ensures that cells within the same column remain closely connected in the topology, making it easier for the model to perform aggregation operations like summing or filtering a column. For complex layouts featuring merged cells or nested headers, Fei Wang et al. utilize multi-granular graphs in Retrieving Complex Tables with Multi-Granular Graph Representation Learning to map the exact semantic links between a single data point and multiple hierarchical labels.
Taking this concept to its logical conclusion, Pei Chen introduced a hypergraph framework in HyTrel: Hypergraph-enhanced Tabular Data Representation Learning. In a standard graph, an edge connects exactly two nodes. In a hypergraph, a "hyperedge" can encompass an arbitrary number of nodes simultaneously. By modeling each row and each column as a distinct hyperedge over its constituent cell nodes, HyTrel mathematically guarantees permutation invariance. The model treats the table as a collection of intersecting sets, allowing it to preserve multi-dimensional context without relying on the brittle, sequential ordering of linearization.
Navigating Hierarchical Trees
Not all tables fit neatly into a flat 2D grid. Real-world spreadsheets and financial reports frequently feature hierarchical, multi-layered headers where a single data cell might fall under multiple nested sub-categories. When these complex structures are flattened into Markdown or CSV, the hierarchical parent-child relationships are completely severed.
To address this layout degradation, Zhiruo Wang et al. developed TUTA: Tree-based Transformers for Generally Structured Table Pre-training, which models complex tables as bi-dimensional trees. TUTA discards standard token position embeddings and replaces them with tree-based coordinates that track a cell's depth and lineage within nested headers.
To scale this capability to modern large language models, Haobo Wang et al. introduced the TreeThinker pipeline in RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis. TreeThinker acts as a pre-processing layer that explicitly extracts messy, multi-layered headers and structures them into explicit trees before passing them to the language model. By decoupling the structural extraction from the semantic reasoning, it prevents the structural context degradation that causes sequential models to lose track of data hierarchies.