Beyond Row-Level Prediction: Evaluating and Learning Composition for Reusable Table Representations

Ali Younes, Saeed Ghoorchian, Maximilian Schambach, Georgia Chalvatzaki, Johannes HöhnearXiv 2026 · arXiv

In the world of tabular data, we’ve gotten very good at predicting the next cell. Tabular Foundation Models (TFMs) like TabPFN can look at a few rows and perform impressive in-context classification. But what if you don't want to predict a label? What if you want to find a similar table in a massive data lake, detect a duplicate, or estimate how much two datasets overlap?

For these "corpus-scale" operations, we need a single, reusable embedding vector for the entire table. A new paper, "Beyond Row-Level Prediction: Evaluating and Learning Composition for Reusable Table Representations," argues that our current models are surprisingly bad at this—and offers a self-supervised way to fix it.

TL;DR

The authors demonstrate that top-tier tabular models, while great at row-level prediction, produce poor whole-table embeddings when using standard pooling. By introducing a new benchmark and a self-supervised loss called SCDM, they show that learning how to compose local cell states into a global vector is just as important as the encoder itself, significantly boosting retrieval and consistency.

What Problem Does This Paper Address?

Tables are inherently compositional: they are made of cells, which form rows and columns, which form a schema. Most research evaluates models on row-level prediction (e.g., "given these features, what is the target?"). However, practical data engineering tasks like indexing, deduplication, and retrieval require table-level embeddings.

The authors identify a major gap: a model can be a world-class predictor but a terrible "fingerprinter." If you take two different partial views of the same table, a good embedding model should recognize they are related. Current TFMs, when subjected to simple "mean pooling" (averaging all cell vectors), often fail this basic test.

Methods

The researchers tackled this by first building a rigorous evaluation pipeline based on four "desiderata":

  1. D1: Consistency under partial views (Do overlapping views have similar embeddings?)
  2. D2: Discriminability (Can the embeddings separate tables by identity, semantics, or statistics?)
  3. D3: Robustness (Are they stable against row/column permutations or noise?)
  4. D4: Efficiency (How many tables can we process per second?)
The evaluation pipeline: from data preparation to measuring the four desiderata.

The SCDM Loss

The paper’s key technical contribution is Support-Conditioned Distribution Matching (SCDM). Instead of relying on manual labels, SCDM uses the "Intersection over Union" (IoU) of table partitions as a supervision signal.

If two partitions of a table overlap significantly, the model is trained to make their embeddings similar. It matches the distribution of embedding similarities to the distribution of actual partition overlaps. Crucially, they kept the underlying TFM (ConTextTab) frozen and only trained a lightweight "compositional head" (like an Attention or Gated MLP layer) using this loss.

The SCDM training pipeline: matching embedding similarity to anchor-based IoU signals.

Datasets

The study uses a mix of controlled synthetic data and messy real-world corpora:

  • Synthetic: Four families (Physics formulas, Geometric shapes, Temporal reasoning, and SCM priors) with known generative factors to test if models can recover underlying semantics.
  • Real-world: Tables sampled from CARTE, OpenML-CC18, and OpenML-CTR23, concatenating feature matrices with target columns to simulate realistic benchmark tasks.

Key Insights and Ideas

  • The "Fingerprint" Paradox: Simple, lightweight methods like HashingSchema (hashing column names and types) and HashingText (hashing CSV strings) are incredibly hard to beat. They often outperformed massive foundation models on retrieval tasks.
  • Pooling is the Bottleneck: The failure of TFMs isn't necessarily in their "brains" (the encoder) but in how we "summarize" their thoughts. Standard mean or max pooling loses the structural nuances that define a table.
  • Composition can be Learned: By training a small head with SCDM on top of a frozen encoder, the authors moved ConTextTab from the bottom of the leaderboard to the top tier. This suggests that the information is in the local states; we just weren't extracting it correctly.
  • Robustness varies wildly: While most models handle row/column shuffling well, value-sensitive models (like text-based ones) struggle more with cell-level noise.

Key Results

The impact of learning the compositional rule was dramatic, especially on synthetic data:

  • Consistency (D1): The Spearman correlation between embedding similarity and partition overlap jumped from a measly 0.04 (fixed pooling) to between 0.43 and 0.50 (learned SCDM head).
  • Retrieval: In a parent-table retrieval task (finding the source table given a small snippet), the learned heads significantly outperformed fixed pooling, though they still trailed the best specialized fingerprints on real-world data.
  • Efficiency: The learned heads added negligible overhead, staying within 5% of the throughput of simple mean pooling.
Summary of metrics across synthetic data, showing the performance of various embedders.

Limitations and Future Work

The authors are transparent about several constraints:

  • Similarity Metric: The study focuses exclusively on cosine similarity. Other metrics or learned distance functions might yield different rankings.
  • Closed-World Retrieval: The retrieval tests used a corpus of 100 tables; scaling this to "open-world" web-scale retrieval remains an open challenge.
  • Encoder-Composition Interaction: While they showed that learning a head helps a frozen encoder, they haven't yet explored training the encoder and the compositional rule together from scratch.

This paper serves as a wake-up call for the tabular ML community: if we want foundation models to be truly "reusable" for data engineering, we need to stop obsessing solely over row-level accuracy and start learning how to compose the whole picture.

Summary updated

Abstract

Tables are compositional, with content spread across a schema, cell values, rows, and columns. Large-scale operations such as retrieval, overlap estimation, deduplication, and data curation need these local signals composed into one reusable embedding vector per table. Table representation methods, however, are mostly evaluated through row-level prediction or supervised downstream tasks, which do not measure table-level embedding quality directly. We introduce a benchmark that evaluates table-level embeddings through four desiderata: consistency under partial views, discriminability across label granularities, robustness to benign perturbations, and efficiency. Across twelve methods on synthetic and real corpora, lightweight schema- and text-based fingerprints are among the strongest. The sharpest contrast comes from two tabular foundation models, TabPFN and ConTextTab: although both are strong row-level predictors, their states, composed by mean, max, or three further parameter-free pooling rules, trail the strongest fingerprints. We therefore ask whether the gap lies in the encoder or in the compositional rule. Keeping ConTextTab frozen, we learn only the compositional rule over its cell states, using support-conditioned distribution matching (SCDM), a self-supervised loss we propose that matches the distribution of embedding similarities to the distribution of partition overlaps within and across tables. Across all fifteen head configurations, the learned rule raises the Spearman correlation between embedding similarity and partial-view overlap from 0.04 to 0.43-0.50 on synthetic data and improves triplet ranking, clustering, and retrieval, with smaller but systematic gains on real data. A downstream parent-table retrieval task confirms these findings. How local states are composed, and with what objective, substantially affects the quality of table-level embeddings.
BibTeX
@misc{younes2026beyond,
  author = {Ali Younes and Saeed Ghoorchian and Maximilian Schambach and Georgia Chalvatzaki and Johannes Höhne},
  title = {{Beyond Row-Level Prediction: Evaluating and Learning Composition for Reusable Table Representations}},
  year = {2026},
  eprint = {2611.14697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14697},
}
Figure