TL;DR
This paper introduces an automated system for generating effective prompts for Large Language Models (LLMs) on tabular data tasks. By employing Reinforcement Learning for column selection and a novel cell-level similarity metric for few-shot example selection, the system significantly improves performance on tasks like data imputation and error detection, without requiring extensive task-specific training.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable capabilities in various natural language tasks, often through carefully crafted prompts and in-context learning. However, applying LLMs to structured tabular data presents unique challenges. Tabular data has a distinct format, requires specific column considerations, and often involves a large number of columns, making manual prompt engineering a complex and time-consuming process. Existing methods for tabular data tasks often rely on manual selection of relevant columns and few-shot examples, which is impractical for large datasets. This paper aims to automate this prompt generation process to make LLMs more effective and efficient for tabular data tasks.
Methods
The proposed system consists of three main modules:
- Reinforcement Learning-based Column Selection (RLCS): This module uses a Reinforcement Learning agent, specifically Soft Q-Learning, to learn an optimal sequence of columns to include in the prompt. The RL agent is trained using a Markov Decision Process (MDP) framework where states represent the dataset description and actions are selecting column names. The reward function is designed to incentivize accurate task performance.
- Cell-Level Similarity based Few-shot Selection (CLFS): This novel approach addresses the selection of few-shot examples. Instead of treating a row as a sentence (as in natural language methods), CLFS embeds each cell independently and computes similarity based on the average of cell-wise similarities. This is designed to better preserve the semantic meaning of tabular data.
- Build Prompt Module: This module combines the selected columns from RLCS and the few-shot examples from CLFS into a predefined prompt template, which is then fed to the target LLM.
Datasets
The authors evaluated their system on 66 datasets across three downstream tasks: Data Imputation (DI), Error Detection (ED), and Entity Matching (EM). These datasets were sourced from platforms like Kaggle and Open ML and varied in size and domain. No new benchmarks were introduced; rather, existing datasets were utilized to demonstrate the system's generalizability.
Key Insights and Ideas
- Column selection and ordering are crucial: The paper empirically demonstrates that not only selecting the right columns but also arranging them in a specific order significantly impacts LLM performance on tabular tasks.
- Cell-level similarity outperforms NL-based methods: The novel CLFS approach, which considers individual cell semantics, proves more effective for tabular data than traditional natural language-inspired similarity measures for few-shot example selection.
- Automation reduces manual effort: The RL-based column selection automates a previously manual and cumbersome process, making prompt engineering more scalable.
- Generalizability across LLMs and tasks: The proposed system shows consistent improvements across different LLMs (Google flan-t5-xxl and Mixtral 8x7B) and various tabular data tasks.
Key Results
The proposed system, RLCS-CLFS, consistently outperformed baseline methods and manual selection approaches across the evaluated tasks and datasets. For instance, in Data Imputation, the system achieved significantly higher accuracy compared to baselines. The paper reports that the RL-based approach can identify optimal column sequences, and the CLFS method shows improved performance over natural language-based few-shot selection. Specific performance gains are detailed in Tables 1, 3, 4, and 5, showing substantial improvements in accuracy and F1-macro scores for various datasets and tasks. For example, on the AMTRAK dataset for Data Imputation, the system achieved an accuracy of 0.98 with flan-t5-xxl, compared to a baseline of 0.57.
Limitations and Future Work
The authors acknowledge that manual column selection and sequencing are cumbersome. While their system automates this, further research could explore:
- Expanding the auto-prompt generation system to a wider range of row-level downstream tasks.
- Developing a unified model for both column selection and sequencing across diverse tabular datasets.
- Identifying and optimizing other prompt components for tabular tasks.
Summary updated