Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation

Han-Jia Ye, Qi-Le Zhou, De-Chuan Zhan

Abstract

Tabular data is prevalent across various machine learning domains. Yet, the inherent heterogeneities in attribute and class spaces across different tabular datasets hinder the effective sharing of knowledge, limiting a tabular model to benefit from other datasets. In this paper, we propose Tabular data Pre-Training via Meta-representation (TabPTM), which allows one tabular model pre-training on a set of heterogeneous datasets. Then, this pre-trained model can be directly applied to unseen datasets that have diverse attributes and classes without additional training. Specifically, TabPTM represents an instance through its distance to a fixed number of prototypes, thereby standardizing heterogeneous tabular datasets. A deep neural network is then trained to associate these meta-representations with dataset-specific classification confidences, endowing TabPTM with the ability of training-free generalization. Experiments validate that TabPTM achieves promising performance in new datasets, even under few-shot scenarios.

The Challenge of Tabular Heterogeneity

In computer vision and natural language processing, pre-training on massive datasets like ImageNet or Wikipedia provides a foundation that generalizes to new tasks. However, this has been notoriously difficult for tabular data. The core challenge is heterogeneity: tabular datasets lack a shared "vocabulary." In one dataset, a column might represent "Age," while in another, it represents "Blood Pressure." Furthermore, the number of columns (dimensionality) varies wildly. Because there is no consistent semantic meaning or structure across different tables, a model trained on one dataset typically cannot be applied to another without significant re-engineering or re-training.

Comparison of vanilla training versus the proposed pre-training and prediction paradigms in TabPTM.

Methodology: TabPTM

The authors introduce TabPTM (Tabular data Pre-Training via Meta-representation). The key innovation is moving from raw feature space to a neighborhood-based meta-representation, which homogenizes diverse datasets into a single format.

1. Meta-Representation (Φ)

Instead of feeding raw features into a neural network, each instance is represented by its distance to a fixed number ($K$) of its nearest neighbors and their corresponding labels. For classification, an instance is represented by its distance to the $K$ nearest neighbors of each class. For regression, it is represented by the distances and labels of the $K$ nearest neighbors in the entire training set. This creates a fixed-size input regardless of the original table's dimensionality.

2. Adaptive Metric Design

To handle noisy or irrelevant features, TabPTM uses a weighted Minkowski distance. The weights are determined by the Mutual Information (MI) between each attribute and the target label, effectively filtering the importance of columns automatically before calculating neighborhood distances.

3. The Shared Model

Because the meta-representation always has a fixed size, a standard Multi-Layer Perceptron (MLP) can be trained across many different datasets simultaneously. This model learns the "transferable knowledge" of how neighborhood patterns relate to final predictions, rather than learning specific feature meanings.

Key Insights: Relationships as Vocabulary

The paper’s central hypothesis is that while features are not shared across datasets, the relationships between instances are. Neighborhood structures (e.g., "this point is very close to many Class A points") are a universal language for tabular data. By converting raw data into neighborhood distances, TabPTM transforms diverse tasks into a single, homogeneous "local prediction" problem.

Visualization of Meta-Representations (MR) showing clear separation between target and non-target classes across heterogeneous data.

Experimental Results

The study utilized an extensive suite of 101 tabular datasets from the UCI Repository, OpenML, and the TinyBenchmark suite, covering domains like healthcare, finance, and software engineering.

  • Superior Ranking: In few-shot settings (4 to 64 examples per class), TabPTM consistently achieved the best average rank across 65 datasets, outperforming strong baselines like XGBoost, LightGBM, and specialized pre-trained models like TabPFN.
  • Full-Shot Performance: Even with entire datasets available, TabPTM (with minimal fine-tuning) outperformed 12 other methods, including deep learning models like FT-Transformer and TabNet.
  • Efficiency: TabPTM is computationally efficient. Its average runtime for few-shot tasks was roughly 1.35 seconds, comparable to XGBoost and significantly faster than deep learning competitors requiring extensive hyperparameter tuning.
  • Zero-Shot Capability: The model demonstrated strong "direct generalization," performing competitively on new datasets without any gradient updates (fine-tuning).

Conclusion and Future Work

TabPTM offers a novel path for tabular foundation models by focusing on local neighborhood structures. While the primary computational bottleneck is the initial nearest neighbor search, this can be mitigated with approximate nearest neighbor (ANN) techniques. Future directions include exploring more complex architectures like Transformers for the shared top-layer model and investigating more sophisticated ways to integrate auxiliary global information into the meta-representation.

Summary updated

Abstract

Tabular data is prevalent across various machine learning domains. Yet, the inherent heterogeneities in attribute and class spaces across different tabular datasets hinder the effective sharing of knowledge, limiting a tabular model to benefit from other datasets. In this paper, we propose Tabular data Pre-Training via Meta-representation (TabPTM), which allows one tabular model pre-training on a set of heterogeneous datasets. Then, this pre-trained model can be directly applied to unseen datasets that have diverse attributes and classes without additional training. Specifically, TabPTM represents an instance through its distance to a fixed number of prototypes, thereby standardizing heterogeneous tabular datasets. A deep neural network is then trained to associate these meta-representations with dataset-specific classification confidences, endowing TabPTM with the ability of training-free generalization. Experiments validate that TabPTM achieves promising performance in new datasets, even under few-shot scenarios.
BibTeX
@misc{ye2023training,
  author = {Han-Jia Ye and Qi-Le Zhou and De-Chuan Zhan},
  title = {{Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation}},
  year = {2023},
  eprint = {2311.00055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.00055},
}
Figure