Introduction
Translating natural language queries into executable SQL (NL2SQL) has seen significant progress with the advent of Large Language Models (LLMs). However, standard systems often struggle with complex reasoning tasks, such as multi-step arithmetic, hypothetical "what-if" scenarios, and bilingual entity resolution. In Chinese-to-SQL tasks, for instance, entity names are frequently transliterated, creating a mismatch with English-language database schemas.
To address these challenges, we introduce OraPlan–SQL, a planning-centric framework that decomposes the NL2SQL task into a transparent, two-stage process. By utilizing a specialized Planner Agent and a SQL Agent refined through feedback-guided meta-prompting, OraPlan-SQL achieved first place in the 2025 Archer NL2SQL Challenge.
The OraPlan-SQL Framework
OraPlan-SQL moves away from complex multi-agent swarms in favor of a streamlined two-agent architecture:
- The Planner Agent: Instead of generating code directly, this agent produces a stepwise natural language plan. This stage is enhanced by Feedback-Guided Meta-Prompting, where failure cases are analyzed and distilled into corrective guidelines (e.g., ensuring float division for percentages). These guidelines are integrated into the Planner’s system prompt to prevent recurring logic errors.
- The SQL Agent: This agent translates the natural language plan into executable SQL. It utilizes Schema Retrieval to focus on relevant tables and In-Context Learning (ICL) to leverage similar query-SQL examples from training data.
To handle bilingual mismatches, the Planner is instructed to perform Multilingual Entity Linking, generating multiple surface-form variants for entities (e.g., "NYC" vs. "New York City") to ensure the SQL Agent can find the correct database match. Finally, the system employs Plan Diversification and Majority Voting, executing multiple candidate plans and selecting the most frequent result to improve robustness.
Experimental Evaluation
The framework was evaluated on the Archer NL2SQL Evaluation Challenge 2025, a bilingual dataset (English and Chinese) designed to test complex reasoning. Performance was measured using Execution Accuracy (EX)—the correctness of the returned data—and SQL Validity (VA)—the syntactical correctness of the code.
Results and Analysis
OraPlan–SQL secured the top position on the Archer leaderboard, achieving an EX of 54.96% in English and 56.67% in Chinese. Notably, the system maintained a SQL validity rate of over 99%, demonstrating high reliability in code generation.
Key Insights
- Ablation Impact: As shown in the ablation study, the "guidelines" derived from meta-prompting are the most critical component. Removing them caused execution accuracy to drop from 72.12% to 44.23% on the development set.
- The Translation Gap: The authors observed that translating Chinese queries into English prior to planning degrades performance. Direct generation from the native language better preserves the linguistic context required for accurate entity resolution.
- Model Scaling: Performance scales significantly with model capability. While GPT-4o achieved competitive results, the framework reached its peak performance using frontier models like GPT-5, particularly in Chinese tasks where it reached 79.81% EX.
Conclusion and Future Work
OraPlan-SQL demonstrates that a single, highly-optimized Planner agent refined by human-in-the-loop feedback is more effective for complex NL2SQL than high-overhead multi-agent architectures. Future work will focus on exploring the framework's performance with smaller open-source models and developing "gold-standard" plan datasets to further enhance the Planner Agent through In-Context Learning.
Summary updated