Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective

Han-Jia Ye, Qi-Le Zhou, De‐Chuan Zhan, Zhan, De-Chuan, Chao, Wei-Lun

Abstract

Pre-training is prevalent in deep learning for vision and text data, leveraging knowledge from other datasets to enhance downstream tasks. However, for tabular data, the inherent heterogeneity in attribute and label spaces across datasets complicates the learning of shareable knowledge. We propose Tabular data Pre-Training via Meta-representation (TabPTM), aiming to pre-train a general tabular model over diverse datasets. The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels. This "meta-representation" transforms heterogeneous tasks into homogeneous local prediction problems, enabling the model to infer labels (or scores for each label) based on neighborhood information. As a result, the pre-trained TabPTM can be applied directly to new datasets, regardless of their diverse attributes and labels, without further fine-tuning. Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.

Introduction

In computer vision and natural language processing, pre-training is the gold standard. Models learn from massive datasets because those domains share a "vocabulary": pixels represent local color/intensity, and words have consistent semantic meanings. Tabular data, however, remains a challenge for transfer learning. Every table is unique; one might describe heart rates while another details credit scores. Even shared column names like "Age" may have different distributions and relationships to target variables across datasets. This heterogeneity typically forces models to be trained from scratch for every new task.

TabPTM (Tabular data Pre-Training via Meta-representation) addresses this by introducing a way to pre-train a single model on diverse, unrelated tabular datasets. By representing data points through their relationships with their nearest neighbors, TabPTM creates a shared vocabulary across heterogeneous tasks, allowing for state-of-the-art performance on new datasets—often in a zero-shot manner.

Methodology: Meta-Representation (MR)

TabPTM sidesteps the feature-matching problem by focusing on neighborhood embeddings. Instead of feeding raw features into a neural network, TabPTM converts every data instance into a Meta-Representation (MR). For a given row, the MR encodes its distance to its $K$ nearest neighbors in the training set.

  • Classification: The model calculates distances to the $K$ nearest neighbors of each class. If a point is close to many "Class A" neighbors, its MR reflects a high likelihood of belonging to Class A.
  • Regression: The MR includes distances to the $K$ nearest neighbors and their corresponding scalar labels.
Comparison of TabPTM's pre-training paradigms against vanilla training.

To ensure these distances are meaningful across different datasets, the authors use a weighted Minkowski distance. Feature weights are determined by the Mutual Information (MI) between each feature and the target label, ensuring that irrelevant columns do not distort neighborhood relationships. The final architecture is a 3-layer Multi-Layer Perceptron (MLP) trained to map these neighborhood-based MRs to labels across dozens of datasets simultaneously.

Experimental Evaluation

The authors evaluated TabPTM using 101 datasets in total:

  • UCI & OpenML: 72 datasets (36 classification, 36 regression) covering domains from healthcare to software engineering.
  • TinyBenchmark: 29 classification datasets used specifically to test few-shot generalization.

Datasets were split into "pre-training" and "downstream" sets to ensure the model was tested on completely unseen data structures.

Key Insights and Results

The core innovation is that while features are heterogeneous, the geometry of a manifold is universal. "Being close to a neighbor with Label X" carries consistent meaning across domains. By using a fixed $K$, every instance is transformed into a vector of the same size, effectively "homogenizing" the data for a single neural network.

Visualization of Meta-Representations (MR) for target and non-target classes, showing clear separability in the shared space.

Performance Highlights

  • Few-Shot Dominance: In scenarios with 4 to 64 "shots" per class, TabPTM consistently achieved the highest average rank compared to strong baselines like XGBoost, LightGBM, and TabPFN.
  • Full-Shot Performance: With light fine-tuning, TabPTM outperformed state-of-the-art models like TabR and FT-Transformer across 36 tasks.
  • Zero-Shot Transfer: The model can be applied to a brand-new table immediately without any gradient updates, leveraging its learned meta-task of neighborhood-based prediction.

Limitations and Future Work

While effective, TabPTM relies on finding $K$ nearest neighbors, which can be a bottleneck for massive datasets (millions of rows), though approximate search methods can mitigate this. Additionally, representing a point only by its neighbors might discard some global structural information. Future work could explore more complex architectures, like Transformers, to better capture patterns within the meta-representations.

Summary updated

Abstract

Pre-training is prevalent in deep learning for vision and text data, leveraging knowledge from other datasets to enhance downstream tasks. However, for tabular data, the inherent heterogeneity in attribute and label spaces across datasets complicates the learning of shareable knowledge. We propose Tabular data Pre-Training via Meta-representation (TabPTM), aiming to pre-train a general tabular model over diverse datasets. The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels. This ''meta-representation'' transforms heterogeneous tasks into homogeneous local prediction problems, enabling the model to infer labels (or scores for each label) based on neighborhood information. As a result, the pre-trained TabPTM can be applied directly to new datasets, regardless of their diverse attributes and labels, without further fine-tuning. Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
BibTeX
@misc{ye2023rethinking,
  author = {Han-Jia Ye and Qi-Le Zhou and De‐Chuan Zhan and Zhan, De-Chuan and Chao, Wei-Lun},
  title = {{Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective}},
  year = {2023},
  eprint = {2311.00055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.00055},
}
Figure