TL;DR
R$^3$-SQL is a novel Text-to-SQL framework that significantly boosts accuracy by addressing two major flaws in current generation-and-ranking pipelines: functional inconsistency (scoring identical execution results differently) and bounded recall (failing when the correct query isn't generated initially). By grouping candidates by execution results and using an LLM agent to selectively resample missing queries, it achieves a new state-of-the-art execution accuracy of 75.03% on the BIRD-dev benchmark for models with disclosed sizes.
What Problem Does This Paper Address?
Most modern Text-to-SQL systems use a "generate-then-rank" approach: a Large Language Model (LLM) generates a pool of candidate SQL queries, and a separate ranking model picks the best one. While effective, this paradigm suffers from two major limitations:
- Functional Inconsistency: SQL queries can look completely different textually (different aliases, join orders, or styles) but produce the exact same execution result. Standard rankers evaluate queries based on surface text, often assigning wildly different scores to functionally identical queries.
- Bounded Recall: Rankers assume that the correct SQL query already exists within the generated pool. If the LLM generator fails to produce the correct query in its initial attempt, the ranker has no way to recover the right answer.
As shown in the figure below, traditional pointwise and listwise rankers can easily rank incorrect candidates first due to superficial textual features, while simple majority voting can be fooled if a large group of incorrect queries happens to produce the same wrong output.
Methods
The authors introduce R$^3$-SQL, a framework that unifies ranking rewards and candidate resampling into a two-phase "exploration-exploitation" process:
- Groupwise Ranking (Exploitation): Instead of scoring queries individually, R$^3$-SQL executes all candidates and groups them by their execution results. It then ranks these groups using a combination of two signals:
- Cross-Group Preference: A pairwise ranker compares candidates across different groups to establish relative preferences. This ranker is trained using Group Relative Policy Optimization (GRPO) with a specialized position-consistency objective to eliminate order bias.
- Pointwise-Group Utility: An order-independent pointwise ranker scores individual candidate quality, which is combined with group size to act as a tie-breaker when pairwise margins are too close.
- Agentic Resampling (Exploration): To overcome bounded recall, an LLM agent acts as a "gatekeeper." It audits the initial candidate pool and predicts whether a correct query is present. If it detects that the correct SQL is likely missing, it discards the pool and triggers a broader resampling phase (generating up to 1,024 candidates) before pruning them back down.
Datasets
The authors evaluated R$^3$-SQL across five prominent Text-to-SQL benchmarks:
- BIRD (BIRD-dev): A large-scale, database-grounded benchmark emphasizing real-world database complexities.
- Spider (Spider-test) & Spider-DK: Widely used cross-domain benchmarks.
- EHRSQL: A domain-specific benchmark tailored to electronic health records.
- ScienceBenchmark: A domain-specific benchmark focused on scientific data.
- Spider2.0-SQLite: Used to validate performance within realistic agentic workflows.
Key Insights and Ideas
- Execution Semantics > Syntax: Evaluating queries at the execution level completely neutralizes the superficial textual noise that typically confuses language models.
- Size Isn't Everything: While previous methods grouped queries by execution, they relied on simple majority voting (picking the largest group). R$^3$-SQL proves that a small, correct group can be identified over a larger, incorrect group by combining learned pairwise preferences with pointwise anchors.
- Selective Resampling Beats Blind Generation: Blindly increasing sample sizes or merging candidate pools introduces excessive noise that distracts rankers. R$^3$-SQL's agentic gatekeeper achieves high precision (93.27%), meaning it only triggers expensive resampling when truly necessary (about 37% of the time), saving computational overhead while maximizing accuracy.
Key Results
- State-of-the-Art Performance: R$^3$-SQL achieved 75.03% execution accuracy (EX) on BIRD-dev, establishing a new state-of-the-art among methods using models with disclosed sizes (32B base model).
- Consistent Gains: It outperformed all baselines across all five benchmarks, breaking the 70% average barrier with an overall average EX of 70.65%.
- Ablation Validation: Removing agentic resampling caused a significant performance drop (from 75.03% to 74.25%), confirming its role in mitigating bounded recall. Furthermore, the position-consistency objective improved input-order robustness by +11.89 percentage points.
Limitations and Future Work
The primary limitation of R$^3$-SQL is its reliance on a supervised pointwise ranker fine-tuned on a specific dataset (BIRD). When evaluated on out-of-domain benchmarks (like EHRSQL or ScienceBenchmark), this pointwise ranker introduced domain-specific noise, slightly bottlenecking the otherwise highly generalized listwise ranker.
The authors note that integrating a more domain-generalized pointwise ranker is a promising direction to unlock even stronger cross-domain performance.
Summary updated