Introduction
Tabular data is fundamental to data science, yet it often lacks the rich context necessary for high-accuracy predictive modeling. While traditional feature engineering (FE) and external enrichment of datasets (EED) provide some relief, they typically rely on structured sources like relational databases. This leaves a vast ocean of unstructured data—such as Wikipedia articles and corporate reports—largely untapped. Furthermore, training deep learning models on small tabular datasets remains a significant challenge.
The FeSTE (Few-Shot Transformer-based Enrichment) framework addresses these hurdles by extracting features from unstructured text and reformulating classification tasks to enable cross-dataset learning.
Methodology
The FeSTE framework operates through a multi-stage pipeline designed to bridge the gap between tabular entities and textual knowledge.
The process consists of three primary phases:
- Entity Linking: Entities within the table (e.g., university names) are linked to Wikipedia pages. The system retrieves the page abstract to serve as contextual text for the record.
- Dataset Task Reformulation: To allow a single model to learn from diverse datasets with varying class structures, FeSTE transforms N-class classification into a binary sentence-pair problem.
- Fine-Tuning and Feature Generation: The model is pre-trained on multiple datasets and then fine-tuned on the target dataset. It generates new features based on the probability that the linked text corresponds to each possible class.
The "Secret Sauce": Task Reformulation
Standard classification models are limited by fixed output layers. FeSTE bypasses this by asking a binary question: "Is this text related to Class X?" The input is structured as [CLS] <Abstract> [SEP] <Possible Class Name> [SEP]. This allows the model to treat classification as a relevance task.
By converting the problem into a relevance task, the model can be trained simultaneously on unrelated datasets (e.g., sports and media), significantly improving its performance in few-shot scenarios.
Experimental Evaluation
The researchers evaluated FeSTE across 17 diverse classification datasets from sources like Kaggle and OpenML. These datasets covered domains such as sports (Baseball), media (Netflix, Anime), and finance (S&P 500), with sizes ranging from 35 to over 4,000 records.
Key Results
- Performance Gains: Integrating FeSTE-generated features into original datasets led to an average accuracy increase of 9.2%.
- Few-Shot Efficacy: On the "Zoo" dataset, FeSTE improved AUC by up to 37% compared to baselines, demonstrating its strength when data is scarce.
- Feature Quality: Even without original features, FeSTE-derived features alone achieved a respectable average AUC of 0.664, proving the generated features carry substantial signal.
Discussion and Limitations
While FeSTE provides a powerful tool for data enrichment, its effectiveness depends on the relevance of the external text. In highly specialized domains where Wikipedia abstracts may not explicitly cover the target labels (e.g., specific economic indicators), the signal can be weak. Additionally, the reformulation process increases computational overhead as the number of sentence pairs scales with the number of classes.
Future work will focus on more sophisticated entity linking and extending the framework to handle regression tasks.
Summary updated