TL;DR
This paper introduces TST-LLM, a framework that uses Large Language Models (LLMs) to transform "task-agnostic" self-supervised learning into a "task-specific" process. By leveraging an LLM's prior knowledge to discover relevant features from text descriptions, the model creates custom pre-text tasks that significantly outperform traditional methods across 22 benchmark datasets.
The Problem: Task Mismatch in Tabular Data
In the world of machine learning, self-supervised learning (SSL) is a superpower: it allows models to learn from vast amounts of unlabeled data. Usually, this is done via "pre-text tasks," like masking a word in a sentence or rotating an image and asking the model to predict the change.
However, in tabular data (the rows and columns found in Excel or SQL), these tasks are often "task-agnostic." For example, a model might learn to reconstruct a masked "Height" column. But if the ultimate goal is to predict heart disease, the model might accidentally discard the crucial correlation between height and weight during training. This "task mismatch" means the learned representations aren't as useful as they could be for the actual goal. The authors aim to bridge this gap by making the pre-training phase aware of the downstream task's objective using metadata.
Methods: The TST-LLM Framework
The authors propose TST-LLM (Task-specific Self-supervised Tabular learning with LLMs), which operates in two main stages to align pre-training with downstream goals.
- LLM-Guided Feature Discovery: Instead of random masking, the system feeds the LLM the natural language description of the task (e.g., "Predict if a patient has diabetes") and the metadata of the columns. The LLM then "engineers" new, informative features (e.g., calculating BMI from height and weight). It does this over multiple trials, with "diversity enforcement" to ensure it doesn't just suggest the same features repeatedly.
- Representation Learning: These LLM-discovered features are treated as ground-truth labels for a pre-text task. The model uses Multi-task Supervised Contrastive Learning (SCL). Essentially, it trains an encoder to pull data samples with similar LLM-generated features closer together in the embedding space and push dissimilar ones apart.
To keep things efficient, the framework includes a Feature Selection step using entropy filtering (to remove uninformative features) and Cramer’s V (to ensure the selected features aren't redundant).
Datasets and Evaluation
The researchers evaluated TST-LLM on 22 benchmark tabular datasets, covering binary classification, multi-class classification, and regression. These included:
- Standard Benchmarks: Adult, Bank, Credit-g, Diabetes, Heart, and Housing.
- Synthetic Benchmarks: Sequence-type and Solution-mix, specifically chosen to ensure the LLM hadn't seen the data during its own training.
- Additional Validation: 8 datasets from the OpenML-CC18 suite.
Key Insights and Ideas
- LLMs as Domain Experts: The core innovation is realizing that LLMs possess enough world knowledge to act as expert feature engineers. An LLM knows that "capital gain minus capital loss" is a vital indicator for income, even if the raw data doesn't explicitly say so.
- Alignment is Everything: By using these "expert" features as labels for pre-training, the model's internal representation becomes naturally aligned with the logic needed for the final task.
- Information Boost: The authors found that LLM-discovered features often have significantly higher Mutual Information with the actual target labels than the original raw features do.
- Simplicity Wins: TST-LLM doesn't require fine-tuning the LLM itself; it only uses prompt engineering, making it lightweight and compatible with any off-the-shelf LLM API (like GPT-3.5 or Llama-3).
Key Results
- Dominant Performance: Against nine contemporary self-supervised baselines, TST-LLM achieved a win ratio of 95% when using a linear evaluator and 81% with a non-parametric (k-NN) classifier.
- Beating Supervised Models: Surprisingly, this unsupervised framework often performed comparably to, or even better than, popular supervised tabular models like XGBoost and LightGBM on several datasets.
- Robustness: The method proved effective across different LLM backbones, with larger models (like Llama-3-70B) generally providing better feature discovery than smaller ones.
- Efficiency: Despite the LLM overhead, the total training time remains comparable to other complex SSL methods because the feature selection process keeps the multi-task learning stage lean.
Limitations and Future Work
- Metadata Dependency: The method relies on having clear natural language descriptions for the task and the features. If your columns are named
Col_1,Col_2, etc., without descriptions, the LLM cannot help. - LLM "Blind Spots": If the task involves a domain the LLM is unfamiliar with, the discovered features might be suboptimal.
- Future Directions: The authors suggest moving toward semi-supervised learning, where a small amount of labeled data could be combined with LLM-guided discovery to further refine the pre-text tasks. They also suggest integrating traditional SSL objectives (like reconstruction) alongside the task-specific ones.
Summary updated