Bidirectional Attention for SQL Generation

Huilin Gao, Tong Guo, Fan Wang, Chao Ma2019 · DOI

TL;DR

This paper tackles the challenging problem of converting natural language questions into SQL queries for databases. The authors introduce a novel model that leverages bidirectional attention and character-level CNN embeddings to better understand the complex relationships between questions and database schemas, achieving state-of-the-art results on the WikiSQL dataset.

What Problem Does This Paper Address?

The core problem is Natural Language to SQL (NL2SQL) generation. Imagine you have a vast database filled with structured information, and you want to ask questions about it using everyday language. The challenge is to automatically translate these natural language questions into precise SQL queries that the database can understand and execute. This is difficult because natural language is inherently ambiguous, and the structure of SQL queries can be complex, involving various clauses, conditions, and aggregations. Existing methods often struggle with accurately modeling the intricate interactions between the question's semantics and the database's schema (columns, tables, etc.), especially as the vocabulary size grows with the number of entities in the database.

Methods

The authors propose a modular approach, breaking down the SQL generation process into three main sub-modules:

  1. COLUMN-SELECT module: This module predicts the column(s) to be used in the SELECT clause of the SQL query.
  2. AGGREGATOR-SELECT module: This module determines the aggregation function (e.g., MAX, MIN, COUNT) to be applied to the selected column.
  3. WHERE module: This is the most complex part, responsible for generating the WHERE clause conditions. It's further broken down into predicting the number of conditions, the column slots, the operator slots (e.g., =, >, <), and the value slots.

Key novelties in their approach include:

  • Bidirectional Attention: Inspired by machine comprehension tasks, they employ a bidirectional attention mechanism. This allows the model to simultaneously consider how words in the question relate to words in the column names, and vice-versa, capturing richer contextual information.
  • Character-level Embedding with CNNs: To handle the large vocabulary and potential out-of-vocabulary words, they use character-level embeddings combined with Convolutional Neural Networks (CNNs). This allows the model to learn representations of words based on their sub-word (character) composition, making it more robust to variations and misspellings.
  • Modular Design: The decomposition into specialized modules for SELECT, AGGREGATOR, and WHERE clauses allows for tailored network architectures and loss functions for each part of the SQL query.

Datasets

The primary dataset used for training and evaluation is WikiSQL. This dataset consists of 80,654 hand-annotated examples, each comprising a natural language question, a corresponding database table (from Wikipedia), and its correct SQL query. The paper notes that the SQL queries in WikiSQL have a fixed structure: SELECT $COLUMN1 [$AGG] FROM TABLE WHERE $COLUMN2 EQUALS $VALUE1 [AND $COLUMN3 EQUALS $VALUE2], with 0-4 conditions in the WHERE clause.

Key Insights and Ideas

  • Bidirectional attention is crucial for understanding question-column relationships: The paper highlights that bidirectional attention significantly improves performance, particularly for the challenging WHERE clause generation. This suggests that a two-way understanding of how question words map to column names (and vice-versa) is essential for accurate SQL generation.
  • Character-level embeddings enhance robustness: Using CNNs on character-level embeddings provides a significant boost to the COLUMN-SELECT module. This indicates that representing words as compositions of characters makes the model more resilient to variations in word forms and potentially out-of-vocabulary words, a common issue in large-scale datasets.
  • Modular design simplifies a complex task: Breaking down the SQL generation into distinct modules (SELECT, AGGREGATOR, WHERE) allows for more focused learning and better performance on each component. This is a common strategy in complex sequence generation tasks.
  • "Order-matters" problem in SQL generation: The paper acknowledges that traditional sequence-to-sequence models can struggle with the "order-matters" problem, where the order of conditions in a WHERE clause might not affect the query's execution but can confuse the model. Their modular approach and attention mechanisms seem to mitigate this.

Key Results

The paper reports achieving state-of-the-art results on the WikiSQL dataset. Specifically:

  • Overall Result Match: Their model achieved a "result match" accuracy of 71.1% on the test set, which compares the execution result of the generated SQL with the ground truth. This is an improvement over baselines like Seq2SQL (62.1%) and SQLNet (69.8%).
  • Query String Match: The "query string match" accuracy (comparing the generated SQL string exactly to the ground truth) was 64.1% on the test set, also outperforming baselines.
  • Module-wise Improvements: The analysis shows that bi-attention mechanisms improved the WHERE clause accuracy by 3-3.5%, while CNN-based character-level embedding improved the COLUMN-SELECT clause accuracy by around 2%.

Limitations and Future Work

  • Fixed SQL Structure: The current model is designed for a specific, relatively simple SQL structure found in WikiSQL. The authors acknowledge that extending this to more complex SQL constructs like ORDER BY, GROUP BY, or JOIN clauses would be more challenging.
  • Masking Technique: They observed that random initialization of out-of-vocabulary words didn't improve results. They suggest that a "mask technique," which assigns minimal values to rare words to prevent them from participating in activation functions, could be a promising future direction.
  • Generalization: While state-of-the-art on WikiSQL, the model's performance on databases with different schemas or more complex natural language queries is not explicitly discussed.
  • Future Work: The authors suggest that the technique of NL2SQL can be applied to other semantic parsing tasks like Knowledge Graph querying, indicating a broader applicability and a rich area for future research.

Summary updated

Abstract

Generating structural query language (SQL) queries from natural language is a long-standing open problem. Answering a natural language question about a database table requires modeling complex interactions between the columns of the table and the question. It has been attracting considerable interest recently and driven by the explosive development of deep learning techniques. In this paper, we apply the sketch-based approach or synthesizing way to solve this problem. Based on the structure of SQL queries, we break down the model to three sub-modules and design specific deep neural networks for each of them. We employ the bidirectional attention mechanisms and character-level embedding with convolutional neural networks (CNNs) to improve the result. Experimental evaluations show that our model achieves the state-of-the-art results in WikiSQL dataset.
BibTeX
@misc{gao2019bidirectional,
  author = {Huilin Gao and Tong Guo and Fan Wang and Chao Ma},
  title = {{Bidirectional Attention for SQL Generation}},
  year = {2019},
  doi = {10.1109/icccbda.2019.8725626},
}
Figure