TL;DR
This paper introduces AutoLink, a novel agent-based framework that tackles the challenge of schema linking for Text-to-SQL in large-scale industrial databases. By iteratively exploring and expanding a relevant schema subset without needing the full database schema, AutoLink achieves state-of-the-art recall and exceptional scalability, outperforming existing methods on large schemas.
What Problem Does This Paper Address?
Text-to-SQL systems aim to translate natural language questions into executable SQL queries. A critical bottleneck for these systems, especially in industrial settings with massive databases, is providing the LLM with the correct database schema. Feeding the entire schema is impractical due to LLM context window limitations and the introduction of irrelevant "noise" that hinders accurate SQL generation. Schema linking, the process of identifying a relevant subset of schema elements, is crucial. However, current schema linking methods often struggle with:
- High costs: Requiring extensive computation or token usage.
- Scalability: Degrading performance on large databases with thousands of tables and columns.
- Recall-noise trade-off: Difficulty in achieving high recall without introducing too much irrelevant information.
This paper addresses the need for a scalable, high-recall schema linking solution that can efficiently handle large industrial databases.
Methods
AutoLink reframes schema linking as an iterative, agent-driven exploration process. Instead of processing the entire schema upfront, an LLM-powered autonomous agent dynamically discovers and builds the necessary schema subset. This agent interacts with two key environments:
- Database Environment (EDB): Allows the agent to execute SQL queries to explore schema metadata and sample data. This helps in verifying schema sufficiency and identifying missing elements.
- Schema Vector Store Environment (Evs): A pre-built vector database of all schema elements (tables, columns, descriptions). This environment enables efficient semantic search to retrieve relevant schema components based on the user's query.
The agent operates through a multi-turn dialogue, employing a set of specialized actions:
@explore_schema: Interacts with EDB to query schema metadata (e.g., column names, types, keys).@retrieve_schema: Uses Evs for targeted semantic search to find potentially missing schema elements.@verify_schema: Executes a draft SQL query to check if the current linked schema is sufficient to answer the question.@add_schema(Schema Expansion): Commits newly discovered relevant schema elements to the linked schema.@stop_action: Terminates the process when the agent determines enough schema elements have been gathered.
Crucially, the agent operates without direct access to the full database schema in each turn, making it efficient for large databases.
Datasets
The paper evaluates AutoLink on two prominent Text-to-SQL benchmarks:
- Bird-Dev: A dataset with 11 databases, averaging 80 columns each, and 1,543 complex SQL queries.
- Spider 2.0-Lite: Derived from industrial applications, this dataset features larger databases (averaging over 800 columns), more intricate queries, and supports multiple SQL dialects (BigQuery, Snowflake, SQLite). It contains 547 test cases.
Key Insights and Ideas
- Agentic Exploration is Key: The core innovation is treating schema linking as an interactive, exploratory process, mimicking how human database engineers work. This allows for dynamic discovery rather than static selection.
- Decoupling Schema Linking from Full Schema Input: AutoLink's ability to function without the entire schema is a major breakthrough for industrial scalability, bypassing LLM context window limitations.
- Synergy of Environments: The combination of direct SQL exploration (EDB) and semantic vector search (Evs) provides a powerful mechanism for both targeted discovery and verification.
- Iterative Schema Expansion: The
@add_schemaaction, when used in conjunction with feedback from other actions, allows for a progressive and efficient build-up of the relevant schema subset. - Strong Correlation between SRR and EX: The paper highlights that higher Strict Recall Rate (SRR) directly translates to better Execution Accuracy (EX) in SQL generation, underscoring the importance of comprehensive schema linking.
Key Results
- State-of-the-Art Recall: AutoLink achieves a strict schema linking recall (SRR) of 97.4% on Bird-Dev and 91.2% on Spider 2.0-Lite.
- Exceptional Scalability: AutoLink maintains high recall and significantly outperforms baseline methods on large databases (over 3,000 columns) where others degrade severely.
- Efficient Token Consumption: AutoLink uses substantially fewer tokens compared to other methods, particularly on larger datasets, making it cost-effective. For example, it uses less than half the tokens of ReFoRCE on Spider 2.0-Lite.
- Competitive Execution Accuracy (EX): Achieves 68.7% EX on Bird-Dev (outperforming CHESS) and 34.9% EX on Spider 2.0-Lite (ranking 2nd on the official leaderboard).
- Ablation Studies: Confirm the importance of each agent action (
@retrieve_schema,@explore_schema,@verify_schema) for achieving high SRR.
Limitations and Future Work
- Limitations: The paper notes that while AutoLink significantly reduces token consumption, it still incurs some overhead due to the iterative nature. The performance of the LLM policy itself is a factor, and the paper mentions using DeepSeek-V3 for AutoLink's policy due to observed instruction following degradation in other models.
- Future Work: The paper suggests that the framework provides a practical and efficient foundation for industrial-scale Text-to-SQL systems. Further exploration of hyperparameter tuning (e.g., initial top-n schema retrieval) and the interplay between schema linking and SQL generation could be avenues for future research. The authors also mention an extended version with full technical appendices will be released on arXiv, which may contain further details.
Summary updated