Domain-Specific Few-Shot Table Prompt Question Answering via Contrastive Exemplar Selection

Tianjin Mo, Qiao Xiao, Hongyi Zhang, Ren Li, Yunsong WuAlgorithms 2024 · DOI

TL;DR

This paper tackles the challenge of domain-specific table question answering (text-to-SQL) with limited data by proposing a novel few-shot approach. They introduce a structured prompt template to standardize model output and a contrastive exemplar selection strategy to help large language models better understand domain-specific queries, achieving state-of-the-art results on electricity and structural inspection datasets.

What Problem Does This Paper Address?

Natural language processing (NLP) has made significant strides in table question answering (text-to-SQL), enabling users to query structured data using natural language. Large Language Models (LLMs) have shown promise in this area, especially with prompt learning, which reduces the need for extensive fine-tuning data. However, applying these general-purpose LLMs to domain-specific tables presents a challenge. These domain-specific tables often have more columns and complex questions, leading LLMs to generate invalid SQL queries. While few-shot learning (using only a few examples) is a solution, effectively selecting the right examples for domain-specific contexts remains difficult. This paper aims to bridge this gap by developing a method that leverages prompt engineering and intelligent exemplar selection for domain-specific few-shot text-to-SQL.

Methods

The proposed approach combines two main strategies:

  • Structured Prompt Template Construction: The authors design a prompt template that standardizes the input and output format for text-to-SQL tasks. This template helps the LLM understand the expected structure of the SQL query, including clauses like SELECT, WHERE, and aggregation functions. By restructuring the input and constraining the output, this method aims to improve the integrity and validity of the generated SQL.
  • Contrastive Exemplar Selection: To address the few-shot learning challenge, they introduce a contrastive exemplar selection strategy. This method uses the LLM itself to score candidate training examples based on their relevance to the test question. It then constructs contrastive learning data by pairing the test question with relevant positive and negative exemplars. This helps the LLM learn domain-specific query patterns and characteristics more effectively, guiding it to retrieve the most pertinent examples.

The core idea is to use a pre-trained LLM (Vicuna-13B) and guide its output through carefully crafted prompts and relevant examples, rather than relying on traditional fine-tuning.

Datasets

The study evaluates its approach on two domain-specific datasets:

  • ESQL: This dataset is sourced from real power energy agencies and covers various dimensions of electricity data, such as time, region, power suppliers, and enterprise profitability. It's characterized by tables with a large number of columns (many having over 20).
  • SMI-SQL: This dataset is derived from industrial databases related to bridge maintenance. It includes information on bridge basics, structural damage, and maintenance recommendations. This dataset also features tables with a significant number of columns, though with a different distribution than ESQL.

Key Insights and Ideas

  • Domain-Specific LLM Adaptation is Crucial: General LLMs struggle with the complexity and nuances of domain-specific tables. Targeted approaches are needed to make them perform well in these specialized areas.
  • Prompt Engineering is More Than Just Formatting: The paper highlights that well-designed prompt templates can significantly improve the structured output of LLMs, making them more reliable for tasks like SQL generation.
  • Intelligent Exemplar Selection Matters: Simply providing random examples isn't enough. The contrastive exemplar selection method, which leverages the LLM's own scoring capabilities, is key to identifying truly relevant examples that guide the model effectively.
  • Few-Shot Learning is Viable for Complex Tasks: By combining prompt engineering with smart exemplar selection, the paper demonstrates that high performance can be achieved even with very limited training data in complex domain-specific text-to-SQL tasks.

Key Results

The proposed approach, termed EPR (Exemplar Prompt Retrieval), significantly outperformed baseline methods (ZERO SHOT, RANDOM, BM25) on both the ESQL and SMI-SQL datasets.

  • Syntax Legality (Syn-SQL): EPR achieved the highest legality scores, reaching 92.6% on ESQL and 95.8% on SMI-SQL, indicating a substantial improvement in generating syntactically correct SQL queries compared to zero-shot (57.7% and 58.6% respectively).
  • Logical Form Accuracy (LF): EPR achieved 23.3% LF on ESQL and 22.4% on SMI-SQL, outperforming other methods.
  • Execution Result Accuracy (EX): EPR achieved 35.8% EX on ESQL and 36.2% on SMI-SQL, demonstrating its ability to generate SQL that produces correct results.

The paper also shows that EPR improved performance across various sub-module tasks (SC, SA, WN, WC, WO, WV) within the SQL generation process.

Limitations and Future Work

  • Sub-module Prediction Errors: The authors acknowledge that the EPR method can still exhibit sub-module prediction errors, particularly with complex WHERE conditions or operators composed of multiple symbols (e.g., !=).
  • Handling Semantic Continuity: The model sometimes struggles with complex questions that involve semantic continuity, leading to fewer WHERE conditions being parsed than actually present.
  • Domain-Specific Nuances: While the approach is domain-specific, the paper notes that the complexity of datasets like ESQL and SMI-SQL can still pose challenges, leading to LF and EX scores that are lower than Syn-SQL.

Future work could focus on improving the model's ability to handle more complex SQL structures and operators, as well as further refining exemplar selection for even more challenging domain-specific scenarios.

Summary updated

BibTeX
@article{mo2024domain,
  author = {Tianjin Mo and Qiao Xiao and Hongyi Zhang and Ren Li and Yunsong Wu},
  title = {{Domain-Specific Few-Shot Table Prompt Question Answering via Contrastive Exemplar Selection}},
  journal = {Algorithms},
  year = {2024},
  doi = {10.3390/a17070278},
}
Figure