TL;DR
LuthorSQL is a fast, local 7B model that acts as a "System 1" gate for Text-to-SQL tasks, predicting query plans in a single forward pass. By handling routine queries locally and escalating only uncertain cases to frontier models like Claude 3.5 Opus, it maintains state-of-the-art accuracy while slashing API costs by up to 34%.
What Problem Does This Paper Address?
Frontier LLM agents (like Claude 3.5 Opus or GPT-4o) are currently the gold standard for translating natural language into SQL. However, they come with three major baggage items: they are slow, expensive (costing tens of thousands of dollars annually for high-volume enterprise use), and privacy-leaking (sending sensitive schema and query data to external APIs).
The authors ask: To what extent can we offload this work to local hardware without losing accuracy? They draw inspiration from Daniel Kahneman’s "System 1 and System 2" framework, designing a fast, intuitive "System 1" (LuthorSQL) to handle the easy stuff and a slow, deliberate "System 2" (a frontier LLM agent) for the hard stuff.
Methods
LuthorSQL moves away from standard token-by-token autoregressive generation. Instead, it uses a non-autoregressive slot-filling approach:
- Structured Plans: The model (a 7B OmniSQL backbone with LoRA adapters) makes a single forward pass to fill "slots" in a bounded query grammar (e.g., projections, filters, joins). This prevents syntax errors and hallucinations.
- Deterministic Compiler: Instead of the model writing complex
JOINclauses, a compiler takes the predicted plan and finds the shortest path between tables using a Steiner tree over the database's foreign-key graph. - Candidate Enumeration & Validation: The model generates up to 32 candidate plans. A second adapter, VALHARD, scores these candidates after they are executed against the database.
- Confidence Cascade: If the margin between the top two candidates is high, LuthorSQL provides the answer. If the margin is low (indicating uncertainty), the query is escalated to the "System 2" LLM agent.
Datasets
The authors utilized several major benchmarks for training and evaluation:
- Training: BIRD, Spider, and SynSQL-2.5M.
- Evaluation: BIRD (dev and hidden test sets), KaggleDBQA, and Spider-Realistic.
- Fresh Benchmark: Crucially, the authors built a new set of 227 questions on databases published after the training cutoffs of major LLMs (July–October 2026) to ensure results weren't just the product of memorization.
Key Insights and Ideas
- One-Pass Efficiency: By predicting all slots in parallel, LuthorSQL achieves a median latency of just 0.84s, significantly faster than autoregressive models that generate SQL token-by-token.
- Zero-Shot Dialect Transfer: Because the model predicts an engine-agnostic plan rather than raw SQL string, it can be transpiled to Snowflake, BigQuery, MySQL, etc., with a performance drop of only ~3 points compared to its native SQLite accuracy.
- The Memorization Trap: Contamination probing revealed that Claude Opus could reproduce 48–92% of public benchmark gold queries verbatim without even seeing the schema. This underscores the importance of the authors' "Fresh" post-cutoff evaluation.
- Tool-Use Overcomes Recall: Despite memorizing benchmarks, the study found that LLM agents maintain high accuracy on renamed schemas and new data because they use database tools to explore, rather than relying solely on internal weights.
Key Results
- Standalone Performance: LuthorSQL reached 68.03% execution accuracy on the BIRD hidden test set. This matches the performance of much larger models (like the 32B Arctic-ExCoT) while using a fraction of the parameters.
- Cost Savings: In a pre-registered confidence cascade, the system handled roughly half of all queries locally, reducing API costs by 33.3% on the "Fresh" dataset at equal accuracy to the standalone frontier model.
- Hardware Efficiency: The model runs in just 9 GB of memory with 8-bit quantization, making it viable for deployment on consumer-grade hardware like an Apple M3 laptop.
Limitations and Future Work
- Grammar Coverage: The current bounded grammar covers about 84.6% of BIRD dev queries. It struggles with complex features like Common Table Expressions (CTEs), window functions, or deeply nested subqueries.
- Validator Tuning: The VALHARD validator was tuned on specific splits, which might lead to slightly optimistic results on those specific distributions.
- Future Directions: The authors suggest developing a dedicated "out-of-grammar" detector to route complex queries to System 2 even before System 1 attempts them, and moving toward native generation in multiple SQL dialects.
Summary updated