In the world of tabular data, we’ve gotten very good at predicting the next cell. Tabular Foundation Models (TFMs) like TabPFN can look at a few rows and perform impressive in-context classification. But what if you don't want to predict a label? What if you want to find a similar table in a massive data lake, detect a duplicate, or estimate how much two datasets overlap?
For these "corpus-scale" operations, we need a single, reusable embedding vector for the entire table. A new paper, "Beyond Row-Level Prediction: Evaluating and Learning Composition for Reusable Table Representations," argues that our current models are surprisingly bad at this—and offers a self-supervised way to fix it.
TL;DR
The authors demonstrate that top-tier tabular models, while great at row-level prediction, produce poor whole-table embeddings when using standard pooling. By introducing a new benchmark and a self-supervised loss called SCDM, they show that learning how to compose local cell states into a global vector is just as important as the encoder itself, significantly boosting retrieval and consistency.
What Problem Does This Paper Address?
Tables are inherently compositional: they are made of cells, which form rows and columns, which form a schema. Most research evaluates models on row-level prediction (e.g., "given these features, what is the target?"). However, practical data engineering tasks like indexing, deduplication, and retrieval require table-level embeddings.
The authors identify a major gap: a model can be a world-class predictor but a terrible "fingerprinter." If you take two different partial views of the same table, a good embedding model should recognize they are related. Current TFMs, when subjected to simple "mean pooling" (averaging all cell vectors), often fail this basic test.
Methods
The researchers tackled this by first building a rigorous evaluation pipeline based on four "desiderata":
- D1: Consistency under partial views (Do overlapping views have similar embeddings?)
- D2: Discriminability (Can the embeddings separate tables by identity, semantics, or statistics?)
- D3: Robustness (Are they stable against row/column permutations or noise?)
- D4: Efficiency (How many tables can we process per second?)
The SCDM Loss
The paper’s key technical contribution is Support-Conditioned Distribution Matching (SCDM). Instead of relying on manual labels, SCDM uses the "Intersection over Union" (IoU) of table partitions as a supervision signal.
If two partitions of a table overlap significantly, the model is trained to make their embeddings similar. It matches the distribution of embedding similarities to the distribution of actual partition overlaps. Crucially, they kept the underlying TFM (ConTextTab) frozen and only trained a lightweight "compositional head" (like an Attention or Gated MLP layer) using this loss.
Datasets
The study uses a mix of controlled synthetic data and messy real-world corpora:
- Synthetic: Four families (Physics formulas, Geometric shapes, Temporal reasoning, and SCM priors) with known generative factors to test if models can recover underlying semantics.
- Real-world: Tables sampled from CARTE, OpenML-CC18, and OpenML-CTR23, concatenating feature matrices with target columns to simulate realistic benchmark tasks.
Key Insights and Ideas
- The "Fingerprint" Paradox: Simple, lightweight methods like
HashingSchema(hashing column names and types) andHashingText(hashing CSV strings) are incredibly hard to beat. They often outperformed massive foundation models on retrieval tasks. - Pooling is the Bottleneck: The failure of TFMs isn't necessarily in their "brains" (the encoder) but in how we "summarize" their thoughts. Standard mean or max pooling loses the structural nuances that define a table.
- Composition can be Learned: By training a small head with SCDM on top of a frozen encoder, the authors moved ConTextTab from the bottom of the leaderboard to the top tier. This suggests that the information is in the local states; we just weren't extracting it correctly.
- Robustness varies wildly: While most models handle row/column shuffling well, value-sensitive models (like text-based ones) struggle more with cell-level noise.
Key Results
The impact of learning the compositional rule was dramatic, especially on synthetic data:
- Consistency (D1): The Spearman correlation between embedding similarity and partition overlap jumped from a measly 0.04 (fixed pooling) to between 0.43 and 0.50 (learned SCDM head).
- Retrieval: In a parent-table retrieval task (finding the source table given a small snippet), the learned heads significantly outperformed fixed pooling, though they still trailed the best specialized fingerprints on real-world data.
- Efficiency: The learned heads added negligible overhead, staying within 5% of the throughput of simple mean pooling.
Limitations and Future Work
The authors are transparent about several constraints:
- Similarity Metric: The study focuses exclusively on cosine similarity. Other metrics or learned distance functions might yield different rankings.
- Closed-World Retrieval: The retrieval tests used a corpus of 100 tables; scaling this to "open-world" web-scale retrieval remains an open challenge.
- Encoder-Composition Interaction: While they showed that learning a head helps a frozen encoder, they haven't yet explored training the encoder and the compositional rule together from scratch.
This paper serves as a wake-up call for the tabular ML community: if we want foundation models to be truly "reusable" for data engineering, we need to stop obsessing solely over row-level accuracy and start learning how to compose the whole picture.
Summary updated