KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

Chia-Hsuan Lee, Oleksandr Polozov, Matthew RichardsonACL 2021

TL;DR

This paper introduces KaggleDBQA, a new benchmark dataset designed to bridge the gap between academic text-to-SQL research and real-world applications. By using actual databases from Kaggle with obscure schemas and natural language questions, they demonstrate that current state-of-the-art parsers struggle significantly, but leveraging database documentation and a few-shot learning approach can dramatically improve performance.

What Problem Does This Paper Address?

The goal of text-to-SQL parsing is to allow users to query relational databases using natural language. While recent large-scale datasets like Spider and WikiSQL have driven progress in zero-shot generalization (i.e., models trained on one set of databases can work on unseen ones), these benchmarks often use simplified or normalized databases and unnaturally phrased questions. This disconnect makes it difficult to deploy current text-to-SQL parsers in real-world scenarios where databases have cryptic schemas, and user questions are more varied and less directly tied to schema terms. The paper aims to create a more realistic evaluation setting that reflects these practical challenges.

Methods

The authors introduce KaggleDBQA, a new dataset and evaluation setting. Key methodological contributions include:

  • Realistic Database Collection: They sourced 8 real-world SQLite databases from Kaggle, which feature abbreviated and obscure column names, domain-specific data types, and original formatting, mirroring industrial databases.
  • Unrestricted Question Annotation: Annotators were tasked with writing natural language questions without direct access to original column names, instead using column descriptions. This aims to create more realistic question phrasing that doesn't rely on direct keyword matching.
  • Few-Shot Evaluation Setting: Instead of strict zero-shot generalization, they propose a few-shot setting where models have access to a small number of in-domain examples (10-20) and database documentation. This is argued to be more representative of how parsers would be adapted in practice.
  • Leveraging Database Documentation: They specifically focus on using column descriptions from the databases to help parsers overcome schema linking challenges, where cryptic column names need to be matched to natural language entities.

Datasets

  • KaggleDBQA: The primary contribution is this new dataset, comprising 8 real-world Kaggle databases and 272 natural language questions with corresponding SQL queries. It's designed to be more challenging than existing benchmarks like Spider and WikiSQL due to its realistic database schemas and question phrasing.
  • Spider: Used for baseline evaluation and comparison. Models were trained on Spider to assess their zero-shot generalization capabilities on KaggleDBQA.

Key Insights and Ideas

  • Real-world schemas are hard: The obscure and abbreviated nature of column names in real-world databases poses a significant challenge for text-to-SQL parsers, even those trained on large datasets like Spider.
  • Natural language is messy: Questions in real-world scenarios are less likely to directly mirror schema terms, making simple keyword matching or direct schema linking insufficient.
  • Documentation is crucial: Database documentation, particularly column descriptions, can act as a vital bridge for parsers to understand cryptic schemas and improve accuracy.
  • Few-shot learning is practical: Assuming a small amount of in-domain data and documentation for adaptation is a more realistic evaluation setting than pure zero-shot learning and yields substantial performance gains.

Key Results

  • SOTA Parsers Struggle: State-of-the-art parsers like RAT-SQL and EditSQL, when evaluated on KaggleDBQA in a zero-shot setting (trained on Spider), achieve significantly lower accuracy (around 13.56% for RAT-SQL) compared to their performance on Spider (around 65.60%).
  • Few-Shot Boost: Moving to a few-shot setting (with 30% in-domain data) for RAT-SQL on KaggleDBQA improves accuracy from 13.56% to 17.96%.
  • Documentation Helps Dramatically: Augmenting RAT-SQL with column descriptions in the few-shot setting, combined with adaptation, boosts accuracy to 26.77%, nearly doubling the zero-shot performance.
  • Normalization Benefits: Normalizing the obscure column names in KaggleDBQA also improves performance, suggesting that both schema clarity and semantic understanding are important.

Limitations and Future Work

  • Limited Multilingual Support: KaggleDBQA currently only includes English questions and databases, requiring translation for multilingual models.
  • Further Documentation Use: The authors suggest future work could explore leveraging other forms of database documentation beyond column descriptions, such as table overviews, sample queries, and data dictionaries.
  • More Complex SQL: While KaggleDBQA has more complex SQL structures than Spider, there's still room for improvement in handling more intricate queries.
  • Value Linking: The paper notes that value linking (matching specific values in questions to database entries) remains a challenge, partly mitigated by value descriptions in documentation.
  • Broader Evaluation: The authors encourage the adoption of their few-shot evaluation regime for established text-to-SQL benchmarks to better reflect real-world deployment scenarios.

Summary updated

BibTeX
@inproceedings{lee2021kaggledbqa,
  author = {Chia-Hsuan Lee and Oleksandr Polozov and Matthew Richardson},
  title = {{KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure