RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, Matthew Richardson

Abstract: Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, Matthew Richardson. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

Introduction

The primary objective of Text-to-SQL research is to enable natural language querying of databases, democratizing data access for non-technical users. However, contemporary models frequently struggle with generalization to unseen schemas. A model trained on a "Hospital" database often fails when applied to a "Car Insurance" database because it lacks a robust mechanism to understand the underlying structural logic of new environments.

The complexity of mapping natural language to SQL, involving both table/column identification (linking) and structural understanding (encoding).

The authors identify two critical hurdles in this domain:

  1. Schema Encoding: Representing the complex web of tables, columns, primary keys, and foreign keys so a model can grasp the database's internal structure.
  2. Schema Linking: Mapping specific words in a natural language question (e.g., "cars") to the correct database elements (e.g., cars_data vs. car_makers), especially when names are ambiguous or do not match exactly.

The RAT-SQL Framework

The core innovation of this work is RAT-SQL (Relation-Aware Transformer). Unlike previous approaches that treated the question and the database schema as separate entities, RAT-SQL treats them as a single, unified graph.

Relation-Aware Self-Attention

Standard Transformers treat inputs as a flat sequence. RAT-SQL modifies the self-attention mechanism to account for the type of relationship between any two elements in the input. If element $i$ is a column and element $j$ is its parent table, the attention calculation is explicitly biased by that specific relational knowledge. This allows the model to "discover" hard database constraints, such as foreign key joins, which standard Transformers often fail to infer implicitly.

The Unified Graph

The model constructs a graph where nodes represent question words, tables, and columns. It defines several categories of edges to facilitate reasoning:

  • Schema Relations: Internal database logic, such as "Column X belongs to Table Y" or "Table A has a foreign key to Table B."
  • Schema Linking: Connections between the question and the schema, including Name-based linking (exact or partial string matches) and Value-based linking (identifying if a question word exists as a literal value within a database column).
RAT-SQL's unified graph approach, representing question tokens and schema elements as nodes connected by specific relational edges.

Experimental Results

The model was primarily evaluated on the Spider benchmark, a large-scale, cross-domain dataset where the test databases do not overlap with the training set, strictly testing generalization capabilities.

  • State-of-the-Art Performance: RAT-SQL achieved 57.2% exact match accuracy, representing an 8.7% absolute improvement over the previous best non-pretrained model. When augmented with BERT embeddings, the model reached 65.6% accuracy.
  • Criticality of Linking: Ablation studies revealed that removing schema linking relations caused accuracy to plummet from approximately 60% to 40%. This confirms that explicit "linking" edges are the most critical component for grounding the model in the specific context of a query.
  • Qualitative Success: Internal alignment visualizations demonstrate that the model correctly learns to ignore "decoy" tables and focuses on the specific tables required for complex joins based on the question's context.

Limitations and Future Work

Despite its success, RAT-SQL faces challenges in Value Decoding, particularly with multi-token constants like specific addresses. Furthermore, roughly 39% of errors involved ambiguous references where the question itself was underspecified. Future work may involve integrating external "common sense" knowledge to resolve queries requiring domain-specific logic (e.g., understanding that "older than 21" implies a specific numerical filter on an age column).

Summary updated

BibTeX
@inproceedings{wang2020rat,
  author = {Bailin Wang and Richard Shin and Xiaodong Liu and Oleksandr Polozov and Matthew Richardson},
  title = {{RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=wgjhTp3Rn-1},
}
Figure