LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks

Sungwon Han, Seungeon Lee, Meeyoung Cha, Sercan O Arik, Jinsung YoonOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

TL;DR

This paper introduces TST-LLM, a framework that uses Large Language Models (LLMs) to transform "task-agnostic" self-supervised learning into a "task-specific" process. By leveraging an LLM's prior knowledge to discover relevant features from text descriptions, the model creates custom pre-text tasks that significantly outperform traditional methods across 22 benchmark datasets.

The Problem: Task Mismatch in Tabular Data

In the world of machine learning, self-supervised learning (SSL) is a superpower: it allows models to learn from vast amounts of unlabeled data. Usually, this is done via "pre-text tasks," like masking a word in a sentence or rotating an image and asking the model to predict the change.

Comparison of task-agnostic vs. task-specific pre-text tasks

However, in tabular data (the rows and columns found in Excel or SQL), these tasks are often "task-agnostic." For example, a model might learn to reconstruct a masked "Height" column. But if the ultimate goal is to predict heart disease, the model might accidentally discard the crucial correlation between height and weight during training. This "task mismatch" means the learned representations aren't as useful as they could be for the actual goal. The authors aim to bridge this gap by making the pre-training phase aware of the downstream task's objective using metadata.

Methods: The TST-LLM Framework

The authors propose TST-LLM (Task-specific Self-supervised Tabular learning with LLMs), which operates in two main stages to align pre-training with downstream goals.

Overall architecture of the TST-LLM framework
  1. LLM-Guided Feature Discovery: Instead of random masking, the system feeds the LLM the natural language description of the task (e.g., "Predict if a patient has diabetes") and the metadata of the columns. The LLM then "engineers" new, informative features (e.g., calculating BMI from height and weight). It does this over multiple trials, with "diversity enforcement" to ensure it doesn't just suggest the same features repeatedly.
  2. Representation Learning: These LLM-discovered features are treated as ground-truth labels for a pre-text task. The model uses Multi-task Supervised Contrastive Learning (SCL). Essentially, it trains an encoder to pull data samples with similar LLM-generated features closer together in the embedding space and push dissimilar ones apart.

To keep things efficient, the framework includes a Feature Selection step using entropy filtering (to remove uninformative features) and Cramer’s V (to ensure the selected features aren't redundant).

Datasets and Evaluation

The researchers evaluated TST-LLM on 22 benchmark tabular datasets, covering binary classification, multi-class classification, and regression. These included:

  • Standard Benchmarks: Adult, Bank, Credit-g, Diabetes, Heart, and Housing.
  • Synthetic Benchmarks: Sequence-type and Solution-mix, specifically chosen to ensure the LLM hadn't seen the data during its own training.
  • Additional Validation: 8 datasets from the OpenML-CC18 suite.

Key Insights and Ideas

  • LLMs as Domain Experts: The core innovation is realizing that LLMs possess enough world knowledge to act as expert feature engineers. An LLM knows that "capital gain minus capital loss" is a vital indicator for income, even if the raw data doesn't explicitly say so.
  • Alignment is Everything: By using these "expert" features as labels for pre-training, the model's internal representation becomes naturally aligned with the logic needed for the final task.
  • Information Boost: The authors found that LLM-discovered features often have significantly higher Mutual Information with the actual target labels than the original raw features do.
  • Simplicity Wins: TST-LLM doesn't require fine-tuning the LLM itself; it only uses prompt engineering, making it lightweight and compatible with any off-the-shelf LLM API (like GPT-3.5 or Llama-3).

Key Results

  • Dominant Performance: Against nine contemporary self-supervised baselines, TST-LLM achieved a win ratio of 95% when using a linear evaluator and 81% with a non-parametric (k-NN) classifier.
  • Beating Supervised Models: Surprisingly, this unsupervised framework often performed comparably to, or even better than, popular supervised tabular models like XGBoost and LightGBM on several datasets.
  • Robustness: The method proved effective across different LLM backbones, with larger models (like Llama-3-70B) generally providing better feature discovery than smaller ones.
  • Efficiency: Despite the LLM overhead, the total training time remains comparable to other complex SSL methods because the feature selection process keeps the multi-task learning stage lean.

Limitations and Future Work

  • Metadata Dependency: The method relies on having clear natural language descriptions for the task and the features. If your columns are named Col_1, Col_2, etc., without descriptions, the LLM cannot help.
  • LLM "Blind Spots": If the task involves a domain the LLM is unfamiliar with, the discovered features might be suboptimal.
  • Future Directions: The authors suggest moving toward semi-supervised learning, where a small amount of labeled data could be combined with LLM-guided discovery to further refine the pre-text tasks. They also suggest integrating traditional SSL objectives (like reconstruction) alongside the task-specific ones.

Summary updated

Abstract

One of the most common approaches for self-supervised representation learning is defining pre-text tasks to learn data representations. Existing works determine pre-text tasks in a "task-agnostic'' way, without considering the forthcoming downstream tasks. This offers an advantage of broad applicability across tasks, but can also lead to a mismatch between task objectives, potentially degrading performance on downstream tasks. In this paper, we introduce TST-LLM, a framework that effectively reduces this mismatch when the natural language-based description of the downstream task is given without any ground-truth labels. TST-LLM instructs the LLM to use the downstream task's description and meta-information of data to discover features relevant to the target task. These discovered features are then treated as ground-truth labels to define "target-specific'' pre-text tasks. TST-LLM consistently outperforms contemporary baselines, such as STUNT and LFR, with win ratios of 95% and 81%, when applied to 22 benchmark tabular datasets, including binary and multi-class classification, and regression tasks.
BibTeX
@misc{han2025llm,
  author = {Sungwon Han and Seungeon Lee and Meeyoung Cha and Sercan O Arik and Jinsung Yoon},
  title = {{LLM-Guided Self-Supervised Tabular Learning With Task-Specific Pre-text Tasks}},
  year = {2025},
  url = {https://openreview.net/forum?id=YO6Je9jOJI},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure