Embedding Individual Table Columns for Resilient SQL Chatbots

Bojan Petrovski, Ignacio Aguado, Andreea Hossmann, Michael Baeriswyl, Claudiu MusatEMNLP 2018

TL;DR

This paper tackles the challenge of making SQL chatbots more robust by moving away from relying on explicit database schema descriptions (like column names). They introduce a novel method called ICE (Individual Column Embeddings) that uses column content to understand tables, and they also create a less biased version of the popular WikiSQL dataset to better evaluate this approach.

What Problem Does This Paper Address?

Natural language interfaces to databases, often called SQL chatbots, aim to let everyday users query relational databases using plain English. The current state-of-the-art often relies on understanding both the user's question and the database's schema, specifically the column names. This creates a significant limitation: if the column names are missing, poorly descriptive (e.g., "col1", "col2"), or change, the chatbot's performance plummets. This reliance on schema knowledge also hinders the chatbot's ability to generalize to new databases. The authors argue that a truly resilient SQL chatbot should be able to infer meaning from the data within the columns rather than just their names, making it accessible even when the schema is unknown or uninformative.

Methods

The core innovation is ICE (Individual Column Embeddings). Instead of encoding column names, ICE creates vector representations for each column based on the content of its cells. The process involves:

  1. Creating a synthetic corpus: For each table column, they concatenate all its cell values to form a "synthetic sentence." They generate multiple shuffled versions of these sentences to capture the idea that cell order within a column doesn't matter semantically.
  2. Training a word2vec model: Using this corpus, they train a word2vec model to learn embeddings for individual words that appear within table columns. The intuition is that words appearing in similar column contexts will have similar embeddings.
  3. Computing column embeddings: For each column, they average the word embeddings of all words in its cells to get a cell embedding, and then take the median of these cell embeddings to represent the entire column.

This ICE representation is then used as a substitute for column header encodings in a sketch-based SQL generation model (specifically, they build upon the SQLNet architecture). This eliminates the need for LSTMs that were previously used to encode column headers, simplifying the model.

Additionally, they address the bias in the WikiSQL dataset. They developed a tool to paraphrase questions in WikiSQL that explicitly mention column names, replacing them with synonyms or more general phrasing. This creates a "column-agnostic" version of the dataset for more realistic evaluation.

Datasets

  • WikiSQL: The primary dataset used for training and evaluation.
  • Column-agnostic WikiSQL: A newly generated dataset created by paraphrasing questions from the original WikiSQL to reduce bias related to explicit column name mentions.

Key Insights and Ideas

  • Column content is a powerful substitute for column names: The paper demonstrates that by focusing on the data within columns, we can achieve robust SQL query generation without needing explicit schema information. This is a significant step towards more generalizable SQL chatbots.
  • Dataset bias is a major hurdle: The WikiSQL dataset's inherent bias towards including column names in questions significantly inflates performance metrics for models that rely on this information. Creating a column-agnostic version is crucial for evaluating true robustness.
  • Simplicity through content-based embeddings: Removing the need to encode column headers simplifies the overall model architecture, potentially leading to faster inference and easier deployment.
  • Resilience to schema changes: The ICE approach makes SQL chatbots inherently more resilient to changes in database schemas, as the model's understanding is tied to the data itself, not its labels.

Key Results

  • When evaluated on the original WikiSQL dataset, their proposed model (SQLNet + ICE) performs comparably to the original SQLNet, achieving accuracies of 89.7% (Aggregation), 92.4% (Selection), and 72.2% (Where-clause) on the dev set.
  • On the column-agnostic WikiSQL dataset, SQLNet's performance drops significantly (e.g., 63.4% for Where-clause), while SQLNet + ICE maintains much higher accuracy (70.1% for Where-clause). This highlights the robustness of the ICE approach.
  • When evaluating on rephrased questions (a subset of the column-agnostic data), SQLNet's accuracy drops further, while SQLNet + ICE shows a much smaller decrease, demonstrating its effectiveness in handling paraphrased queries.

Limitations and Future Work

  • Limitations: The authors acknowledge that the accuracy on the column-agnostic dataset is lower than on the original WikiSQL, particularly for the complex "where clause generation" task. This is expected, as the original dataset's bias provided strong cues. They also note that their paraphrasing method could only rephrase about 20% of the dataset due to a lack of suitable synonyms for all column names.
  • Future Work: The authors suggest extending the system to handle multiple tables (joins), improving performance on more complex operations, and collecting more diverse data to cover a wider range of user utterances and database operations. They also emphasize the need for more varied methods of question collection to reduce dataset bias.

Summary updated

BibTeX
@inproceedings{petrovski2018embedding,
  author = {Bojan Petrovski and Ignacio Aguado and Andreea Hossmann and Michael Baeriswyl and Claudiu Musat},
  title = {{Embedding Individual Table Columns for Resilient SQL Chatbots}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure