Few-Shot Tabular Data Enrichment Using Fine-Tuned Transformer Architectures

Asaf Harari, Gilad Katz

Introduction

Tabular data is fundamental to data science, yet it often lacks the rich context necessary for high-accuracy predictive modeling. While traditional feature engineering (FE) and external enrichment of datasets (EED) provide some relief, they typically rely on structured sources like relational databases. This leaves a vast ocean of unstructured data—such as Wikipedia articles and corporate reports—largely untapped. Furthermore, training deep learning models on small tabular datasets remains a significant challenge.

The FeSTE (Few-Shot Transformer-based Enrichment) framework addresses these hurdles by extracting features from unstructured text and reformulating classification tasks to enable cross-dataset learning.

Methodology

The FeSTE framework operates through a multi-stage pipeline designed to bridge the gap between tabular entities and textual knowledge.

Overview of the FeSTE workflow, from entity linking to feature evaluation.

The process consists of three primary phases:

  1. Entity Linking: Entities within the table (e.g., university names) are linked to Wikipedia pages. The system retrieves the page abstract to serve as contextual text for the record.
  2. Dataset Task Reformulation: To allow a single model to learn from diverse datasets with varying class structures, FeSTE transforms N-class classification into a binary sentence-pair problem.
  3. Fine-Tuning and Feature Generation: The model is pre-trained on multiple datasets and then fine-tuned on the target dataset. It generates new features based on the probability that the linked text corresponds to each possible class.

The "Secret Sauce": Task Reformulation

Standard classification models are limited by fixed output layers. FeSTE bypasses this by asking a binary question: "Is this text related to Class X?" The input is structured as [CLS] <Abstract> [SEP] <Possible Class Name> [SEP]. This allows the model to treat classification as a relevance task.

Task reformulation: converting multi-class labels into binary sentence-pair tuples.

By converting the problem into a relevance task, the model can be trained simultaneously on unrelated datasets (e.g., sports and media), significantly improving its performance in few-shot scenarios.

Experimental Evaluation

The researchers evaluated FeSTE across 17 diverse classification datasets from sources like Kaggle and OpenML. These datasets covered domains such as sports (Baseball), media (Netflix, Anime), and finance (S&P 500), with sizes ranging from 35 to over 4,000 records.

Key Results

  • Performance Gains: Integrating FeSTE-generated features into original datasets led to an average accuracy increase of 9.2%.
  • Few-Shot Efficacy: On the "Zoo" dataset, FeSTE improved AUC by up to 37% compared to baselines, demonstrating its strength when data is scarce.
  • Feature Quality: Even without original features, FeSTE-derived features alone achieved a respectable average AUC of 0.664, proving the generated features carry substantial signal.

Discussion and Limitations

While FeSTE provides a powerful tool for data enrichment, its effectiveness depends on the relevance of the external text. In highly specialized domains where Wikipedia abstracts may not explicitly cover the target labels (e.g., specific economic indicators), the signal can be weak. Additionally, the reformulation process increases computational overhead as the number of sentence pairs scales with the number of classes.

Future work will focus on more sophisticated entity linking and extending the framework to handle regression tasks.

Summary updated

Abstract

The enrichment of tabular datasets using external sources has gained significant attention in recent years. Existing solutions, however, either ignore external unstructured data completely or devise dataset-specific solutions. In this study we proposed Few-Shot Transformer based Enrichment (FeSTE), a generic and robust framework for the enrichment of tabular datasets using unstructured data. By training over multiple datasets, our approach is able to develop generic models that can be applied to additional datasets with minimal training (i.e., few-shot). Our approach is based on an adaptation of BERT, for which we present a novel fine-tuning approach that reformulates the tuples of the datasets as sentences. Our evaluation, conducted on 17 datasets, shows that FeSTE is able to generate high quality features and significantly outperform existing fine-tuning solutions.
BibTeX
@inproceedings{harari2022few,
  author = {Asaf Harari and Gilad Katz},
  title = {{Few-Shot Tabular Data Enrichment Using Fine-Tuned Transformer Architectures}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.111},
}
Figure