TL;DR
This paper tackles the challenge of accurately identifying relevant SQL tables for Text-to-SQL tasks, especially when dealing with heterogeneous data schemas and natural language questions. They introduce a "Reverse Engineering" approach that uses LLMs to generate potential questions for each table, enabling a more direct semantic comparison and significantly improving table acquisition performance.
What Problem Does This Paper Address?
Text-to-SQL, the task of generating SQL queries from natural language questions, relies heavily on first identifying the correct tables within a database that contain the relevant information. This "table acquisition" step is crucial but faces a significant hurdle: the "semantic gap" between natural language questions and the structured, often fragmented, schema names of SQL tables. For instance, a question might ask about "train stations" while the table schema might list fields like "station_name," "location," and "capacity." This mismatch makes it difficult for current models to accurately match questions to tables. Existing methods often struggle with this heterogeneity, leading to suboptimal performance in the downstream Text-to-SQL task.
Methods
The core contribution of this paper is a Reverse Engineering (RE) based optimization approach for table acquisition. Instead of directly matching questions to table schemas, RE generates a "potentially-matched question" (qo) for each SQL table using a Large Language Model (LLM), specifically Qwen2.5-7B-Instruct. This generated question qo is then used to create a more direct, homogeneous semantic comparison with the original input question (q).
This RE approach is integrated into a standard Retriever-Reranker (2R) framework. The Retriever, using BGE-M3 embeddings, now considers both the original question-schema similarity and the question-generated question similarity. The Reranker also takes the generated question as an additional input. This conditional 2R model aims to bridge the semantic gap by allowing the model to compare questions with questions, rather than questions with fragmented schemas.
Datasets
The experiments were conducted on two benchmarks:
- SpiderUnion: A combination of tables from the Spider dataset.
- BIRDUnion: A combination of tables from the BIRD dataset.
Both datasets contain a significant number of questions and associated SQL tables, used for evaluating the table acquisition performance.
Key Insights and Ideas
- Bridging the Semantic Gap with LLM-Generated Questions: The most significant insight is that using an LLM to "reverse engineer" a question for a given table schema can effectively bridge the semantic gap. This allows for a more direct and robust comparison between the original question and the table's implied intent.
- Homogeneous Comparison is Key: By generating a question (qo) for each table, the problem shifts from comparing a natural language question to a structured schema to comparing two natural language questions. This is a more natural task for LLMs and improves relevance estimation.
- Task-Specific Fine-tuning Matters: The paper highlights that fine-tuning the embedding models (like BGE-M3) and the reranker on the specific task of Text-to-SQL table acquisition is crucial for achieving optimal performance.
- Dynamic Weighting for Retriever: The authors propose a dynamic weighting scheme for combining the original question-schema similarity and the question-generated question similarity. This adaptive approach proves more effective than fixed weights across different datasets and models.
Key Results
- Substantial Improvements: The proposed RE approach, when integrated with the 2R baseline, yields substantial improvements in table acquisition metrics like Unbroken Recall (UR@k) and Partial Recall (PR@k) on both SpiderUnion and BirdUnion datasets. For instance, the "Ours" model (RE+Conditional 2R) achieves UR@5 of 93.9 and PR@5 of 97.0 on SpiderUnion, significantly outperforming baselines.
- Competitive Performance: The approach achieves competitive performance compared to strong existing models like CRUSH and MURRE, even when using a smaller baseline model (BGE-M3 0.5B) compared to their larger models.
- Text-to-SQL Enhancement: The improved table acquisition directly benefits the end-to-end Text-to-SQL task, leading to better Execution Accuracy (EX@k) when using LLMs for query generation.
Limitations and Future Work
- One-to-One Generation: The current RE approach generates a question for each table individually. However, a single question might align with multiple SQL tables. The authors suggest that future work should explore multi-table reverse engineering and decomposing questions into subquestions to handle such cases.
- Hybrid Schema and Value Generation: The paper acknowledges that while schema is important, SQL contents (values) also contribute. Future work will involve generating hybrid groups of SQL schemas and values for more comprehensive reverse engineering.
- Graph-Based Reverse Engineering: The authors propose exploring graph-based reverse engineering for table acquisition and Text-to-SQL, which could potentially capture more complex relationships.
Summary updated