In natural language processing, the order of words is everything. "The dog bit the man" and "The man bit the dog" describe two very different scenarios. Consequently, the Transformer architecture—the engine behind modern AI—relies heavily on positional encodings to understand where each word sits in a sequence.
However, tabular data is fundamentally different. In a spreadsheet, if you swap the "Age" column with the "Income" column, or move the 10th row to the 1st position, the underlying information remains identical. This property is known as permutation invariance.
Recent research suggests that the primary obstacle to creating "Foundation Models" for tabular data is this mismatch: we are applying sequence-aware architectures to order-independent data. To solve this, researchers are redesigning Transformers to treat tables as sets rather than sequences.
The Problem with Positional Bias
When we flatten a table into a string to feed it into a Large Language Model (LLM), we introduce an artificial "spatial" relationship. As noted in Large Language Models (LLMs) on Tabular Data: Prediction, Generation, and Understanding - A Survey, LLMs are highly sensitive to the sequence of rows and columns. A model might give a different credit score prediction simply because the "Debt" column appeared before the "Savings" column in the prompt.
Jingfeng Yang and colleagues demonstrated in TableFormer: Robust Transformer Modeling for Table-Text Encoding that standard models can see a 6% drop in accuracy when rows are shuffled. This happens because absolute positional embeddings anchor a cell's meaning to its specific coordinate, rather than its relationship to other features.
Achieving Invariance by Removing the "Where"
The most direct way to build intuition for permutation invariance is to look at what happens when we remove positional encodings entirely.
In Towards Cross-Table Masked Pretraining for Web Data Mining, Chao Ye and his team argue that by omitting these encodings, the Transformer backbone is forced to treat input features as an unordered set. This provides a necessary inductive bias: the model learns that the "meaning" of a feature comes from its content and its relationship to other features, not its index in a list.
Similarly, Maximilian Schambach and Johannes S. Otterbach show in Scaling Experiments in Self-Supervised Cross-Table Representation Learning that this design allows a single model to learn from entirely different datasets simultaneously. Because the model doesn't expect a fixed "schema" or column order, it can focus on the universal statistical patterns shared across diverse tables.
From Sequences to Sets and Graphs
Beyond just removing information, researchers are introducing new structural biases that respect the "grid" nature of tables:
- Relative Structural Bias: Instead of absolute positions (e.g., "this is the 5th token"), TableFormer uses learnable attention biases that only care if two cells are in the same row or column.
- Hypergraph Representations: Pei Chen and Sheng Zha introduced HyTrel, which treats cells as nodes and rows/columns as "hyperedges." This mathematically guarantees that reordering the table doesn't change the model's internal representation.
- Permutation Invariant Pooling: In domains like chemistry, Phuoc-Chung Van Nguyen uses max-pooling across time steps in SynCat to ensure that the order in which reactants are listed doesn't change the predicted reaction outcome.
Generative Robustness
This shift is equally critical for synthetic data generation. If a model is sensitive to order, it might generate high-quality data for one column sequence but "hallucinate" or fail if the columns are rearranged.
Robert Birke and Lydia Y. Chen addressed this in FCT-GAN by using Fourier Transforms. By moving from the spatial domain to the frequency domain, the model captures global correlations between columns that are inherently robust to shuffling. For more complex, heterogeneous data, Hengrui Zhang proposed TABDAR, which uses bi-directional attention and random masking to learn dependencies in an order-agnostic manner.
Why It Matters: The Path to Foundation Models
The ultimate goal of this research direction is to move away from "single-table" models toward Tabular Foundation Models.
Traditional models like Gradient Boosted Decision Trees (GBDTs) are powerful but brittle; they must be retrained from scratch for every new table. By enforcing permutation invariance, models like CTSyn by Xiaofeng Lin and FLAT by Mihaela van der Schaar can generalize across datasets with entirely different feature sets.
When a model treats a table as a set of semantic relationships rather than a fixed string of text, it can finally begin to "reason" about structured data with the same flexibility that LLMs reason about language.