Abstract
RubikSQL is an industrial-grade NL2SQL (Natural Language to SQL) system that treats query generation as a lifelong learning task rather than a one-off translation. By maintaining an "Agentic Knowledge Base" that evolves with user interactions, it successfully tackles the "messy" realities of enterprise data—such as implicit user intents, domain-specific jargon, and massive, complex schemas—achieving state-of-the-art performance on major benchmarks.
The Industrial NL2SQL Challenge
While Large Language Models (LLMs) have revolutionized NL2SQL, a massive gap remains between "benchmark performance" and "industrial utility." In a real-world enterprise, queries are rarely textbook-perfect. RubikSQL identifies four critical hurdles:
- Implicit Intent: Users often expect complex logic (e.g., Year-over-Year comparisons) not explicitly stated in the query.
- Private Domain Knowledge: Companies use non-standard acronyms and custom financial formulas.
- Wide Table Schemas: Databases often contain hundreds of columns requiring complex joins.
- Context Sensitivity: Query meanings change based on data update cycles or user roles.
Unlike benchmarks like Spider that focus on cross-database generalization, RubikSQL focuses on specialization—becoming an expert on a single, complex enterprise ecosystem over time.
Methodology: The Agentic Knowledge Base
The core of RubikSQL is its Agentic Knowledge Base (KB), which functions as a semantic layer decoupling knowledge storage from its utilization. This system is powered by a new standardized format called UKF 1.0 (Unified Knowledge Format).
System Architecture
As shown in the architecture diagram, RubikSQL operates across four distinct layers: Storage, Semantic, Engine, and App. The Unified Knowledge Format (UKF) acts as the critical bridge, ensuring that domain knowledge is accessible regardless of the underlying storage medium (e.g., relational databases, vector stores, or file systems).
Multi-Agent Workflow
The system employs three specialized agents to handle the generation process:
- RAG Agent: Retrieves relevant domain knowledge and schema snippets.
- SQL Gen Agent: Drafts the SQL using Chain-of-Thought (CoT) reasoning.
- SQL Refine Agent: Executes the code in a sandbox, catches errors, and iterates until the query is valid.
Lifelong Learning and Knowledge Distillation
RubikSQL treats the query stream as a source of truth. It mines both labeled and unlabeled user queries to extract "experiences" (CoTs, subqueries, and comments), which are fed back into the KB. To maintain speed and cost-efficiency, a "teacher" model (like DeepSeek-V3/R1) generates high-quality reasoning traces to fine-tune smaller "student" models (like Qwen-32B).
Evaluation and Results
RubikSQL was evaluated on several benchmarks, including the new RubikBench, a large-scale dataset derived from a real multinational car manufacturer's financial database featuring ~5,000 complex queries.
- State-of-the-Art Performance: RubikSQL achieved 77.3% Execution Accuracy (EX) on the BIRD Mini-Dev benchmark, surpassing previous leaders.
- KaggleDBQA: It reached 58.9% EX, a significant jump over existing baselines.
- Efficiency: Through knowledge distillation, a fine-tuned 32B model rivaled the performance of much larger teacher models when equipped with the RubikSQL knowledge base.
Key Insights and Future Work
A standout finding of this research is that only 12% of errors were caused by insufficient LLM reasoning. The vast majority of failures stemmed from missing domain knowledge or misunderstood user intent. This suggests that the primary bottleneck for industrial NL2SQL is knowledge management, not model size.
Future work will focus on deeper user profiling and more sophisticated autonomous search capabilities within the knowledge base to handle increasingly ambiguous requests.
Summary updated