Addressing Limitations of Encoder-Decoder Based Approach to Text-to-SQL

Octavian Popescu, Irene Manotas, Ngoc Phuoc An Vo, Hangu Yeo, Elahe Khorashani, Vadim SheininCOLING 2022

TL;DR

This paper tackles the significant performance drop of Text-to-SQL models on unseen databases. They introduce a hybrid system combining a Disambiguation Dictionary Module (DDM) for schema linkage and a Seed Training Data Augmentation (STDA) technique, leading to substantial accuracy improvements on both known and novel database schemas.

What Problem Does This Paper Address?

The Text-to-SQL task aims to translate natural language questions into executable SQL queries. While models have achieved impressive accuracy on benchmark datasets like Spider, a major challenge arises when these models are applied to databases outside the training distribution. This paper highlights a dramatic performance degradation (often below 20% accuracy) on such "unseen" or "out-of-domain" databases. This limitation severely restricts the practical applicability of current Text-to-SQL systems, as real-world scenarios often involve diverse and novel database schemas. The authors identify that a key reason for this drop is the inability of standard encoder-decoder models to effectively discern relationships between query tokens and specific database schema elements, especially when faced with limited training data for new schemas.

Methods

The proposed solution is a Hybrid System (HS) that enhances the standard encoder-decoder architecture. The core innovations are:

  • Disambiguation Dictionary Module (DDM): This module replaces the default NLP pre-processing. It creates schema-dependent lexical rules and structured data items (DTIs) that explicitly link natural language tokens to specific tables, columns, and their relationships within a given database schema. This provides crucial context to the model.
  • DDM Style Question Rewriting: The DDM augments the input question with explicit references to tables and columns, making the schema information directly accessible to the encoder-decoder.
  • Pseudo-values for Logically Compound Filters: To handle complex SQL constructs that standard models struggle with, the system uses pseudo-values to represent composite filters, simplifying the query structure for the model.
  • Seed Training Data Augmentation (STDA): Recognizing that new databases often have very limited training examples (tens to hundreds), STDA automatically generates more training data by replacing values in existing examples. This is crucial for adapting models to new schemas.
  • Ensemble Technique: Multiple models are trained using data augmented via STDA. These models are then ensembled, and the final SQL query is selected based on a set of hard-coded restrictions derived from the DDM's output, ensuring consistency between values, operators, and the schema.

Datasets

  • Spider Dataset: Used for training the baseline models and as a source for a large portion of the training data.
  • Custom Databases (HR, WH, BI): These are the primary "unseen" databases used for evaluation, representing real-world scenarios where models are expected to perform.
  • Additional Spider Schemas (Pets_1, Dog_kennels): Included for evaluation, specifically chosen because the default system scored poorly on them.
  • Manually Created Temporal Question Corpus: To address the poor representation of temporal questions in Spider, the authors created specific corpora (tmp_1v, tmp_mv) for HR and WH databases.

Key Insights and Ideas

  • Schema Linkage is Paramount: The paper strongly suggests that simply having a large, general-purpose training corpus (like Spider) is insufficient for good performance on new databases. Explicitly linking natural language queries to the specific database schema elements is critical.
  • Data Augmentation is Essential for Low-Resource Scenarios: For new databases with very few training examples, techniques like STDA are vital for creating enough data to train effective models.
  • Ensembling Improves Robustness: Combining predictions from multiple models trained on augmented data helps to overcome individual model weaknesses and leads to more reliable SQL generation.
  • Error Analysis Guides Solutions: The detailed error analysis (EA1-EA6) directly informed the development of the DDM and other components, highlighting specific failure modes of standard encoder-decoder approaches.

Key Results

  • The hybrid system (HS+STDA+Ens) achieved significant improvements over the baseline Valuenet system.
  • For the custom databases (HR, WH, BI), accuracy gains ranged from +6% to +63% compared to the baseline.
  • On average, STDA provided a double-digit percentage improvement for all schemas.
  • The ensemble technique further boosted accuracy by an additional 5-10%.
  • For temporal questions, the hybrid system achieved a good accuracy (e.g., 58% for HR tmp_1v) where the baseline system performed poorly (0% for tmp_mv).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that building a "perfect" training corpus for any new schema is likely impossible.
  • The ensemble's precision is high (>98%), but its recall is moderate (47%), suggesting room for improvement in capturing all correct queries.
  • The EA2 issue of logically compound filters is addressed, but the paper notes it's difficult to fairly compare against systems that don't handle this.
  • Future Work:
    • Further improving training with queries that are not well-represented in the Spider corpus.
  • Developing the methodology for data augmentation.
  • Investigating other systems that can manage compound logical operators.

Summary updated

BibTeX
@inproceedings{popescu2022addressing,
  author = {Octavian Popescu and Irene Manotas and Ngoc Phuoc An Vo and Hangu Yeo and Elahe Khorashani and Vadim Sheinin},
  title = {{Addressing Limitations of Encoder-Decoder Based Approach to Text-to-SQL}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure