Encoding Database Schemas with Relation-Aware Self-Attention for Text-to-SQL Parsers

Richard ShinarXiv 2019 · OpenReview

Abstract

When translating natural language questions into SQL queries to answer questions from a database, we would like our methods to generalize to domains and database schemas outside of the training set. To handle complex questions and database schemas with a neural encoder-decoder paradigm, it is critical to properly encode the schema as part of the input with the question. In this paper, we use relation-aware self-attention within the encoder so that it can reason about how the tables and columns in the provided schema relate to each other and use this information in interpreting the question. We achieve significant gains on the recently-released Spider dataset with 42.94% exact match accuracy, compared to the 18.96% reported in published work.

Introduction

The task of Text-to-SQL parsing aims to bridge the gap between natural language and structured data, allowing users to query databases without specialized knowledge. While early models succeeded on single-table datasets like WikiSQL, they often failed to generalize to "cross-domain" scenarios where the database schema at test time was unseen during training. Real-world databases are complex, featuring multiple tables linked by intricate relationships such as primary and foreign keys. This paper addresses the challenge of cross-domain generalization by arguing that a model must understand the topology of the database schema as deeply as it understands the user's question.

Methodology

The proposed architecture utilizes a specialized encoder-decoder framework designed to be "schema-aware."

Schema as a Graph

Instead of treating the database schema as a flat list of strings, the model represents it as a directed graph. In this graph, nodes represent tables and columns, while edges represent structural relationships. These relationships include "column belongs to table," "column is a primary key," and "column is a foreign key for another table."

The database schema represented as a graph where nodes are tables and columns, and edges represent structural relationships like primary and foreign keys.

Relation-Aware Self-Attention

The core innovation is the modification of the standard Transformer self-attention mechanism to incorporate these graph edges. By embedding edge labels directly into the attention layers, the model explicitly "reasons" about how schema elements connect. This allows question tokens to attend to relevant schema elements and vice versa in a co-encoding process.

Tree-Structured Decoder

To ensure the output is syntactically valid SQL, the system employs a decoder that generates an Abstract Syntax Tree (AST). This approach prevents the generation of invalid SQL strings by following the grammar of the SQL language during the decoding process.

The complete model architecture, featuring a relation-aware encoder that co-processes the question and schema graph, followed by a tree-structured decoder.

Experimental Setup and Results

The model was evaluated on the Spider dataset, a large-scale, complex, and cross-domain Text-to-SQL benchmark. Unlike previous datasets, Spider requires joins across multiple tables and the use of complex clauses like GROUP BY and UNION.

The results demonstrated a significant breakthrough in performance:

  • Proposed Method: 42.94% exact match accuracy.
  • Previous State-of-the-Art (SyntaxSQLNet): 18.96% accuracy.

Ablation studies confirmed that the relation-aware component was the primary driver of these gains. Removing specific relation types caused a sharp decline in accuracy, proving that structural metadata is critical for natural language understanding in database contexts. The model was particularly effective on "Hard" and "Extra Hard" queries, where understanding table joins is essential.

Conclusion

By treating the database schema as a graph and utilizing relation-aware self-attention, this work significantly improves the ability of Text-to-SQL parsers to generalize across domains. While the results on the Spider dataset represent a major leap forward, challenges remain in handling the most complex queries. Future work may involve integrating pre-trained language models like BERT and moving away from heuristic-based reconstruction of JOIN logic toward fully end-to-end neural generation.

Summary updated

Abstract

When translating natural language questions into SQL queries to answer questions from a database, we would like our methods to generalize to domains and database schemas outside of the training set. To handle complex questions and database schemas with a neural encoder-decoder paradigm, it is critical to properly encode the schema as part of the input with the question. In this paper, we use relation-aware self-attention within the encoder so that it can reason about how the tables and columns in the provided schema relate to each other and use this information in interpreting the question. We achieve significant gains on the recently-released Spider dataset with 42.94% exact match accuracy, compared to the 18.96% reported in published work.
BibTeX
@misc{shin2019encoding,
  author = {Richard Shin},
  title = {{Encoding Database Schemas with Relation-Aware Self-Attention for Text-to-SQL Parsers}},
  year = {2019},
  eprint = {1906.11790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.11790},
}
Figure