Introduction
Financial professionals frequently require precise data from relational databases to conduct fund, stock, and macroeconomic analyses. However, a significant barrier exists: the gap between natural language queries and the complex SQL required to retrieve this data. While Large Language Models (LLMs) like GPT-4 have advanced the state of Text-to-SQL, the financial sector presents unique challenges regarding data privacy, operational costs, and extreme schema complexity.
FinSQL is a model-agnostic framework designed to overcome these hurdles. By utilizing a pipeline that features parallel schema linking, LoRA-based "plugin" merging, and execution-free output calibration, FinSQL enables high-performance querying on open-source models that can be deployed locally, ensuring data security while outperforming proprietary models like GPT-4.
The "Wide Table" Challenge in Finance
Standard Text-to-SQL benchmarks like Spider or BIRD often feature relatively simple schemas. In contrast, real-world financial databases are characterized by "wide tables" containing hundreds of columns, often labeled with vague abbreviations. This density creates a massive context window that standard models struggle to process efficiently.
To address this, the authors introduced BULL, a practical financial benchmark collected from Hundsun Technologies Inc. BULL encompasses three domains: Fund, Stock, and Macro Economy. As shown in the database structure, these tables are significantly more complex than those in general-purpose benchmarks, averaging 390 columns per database compared to BIRD's 54.2.
The FinSQL Framework
The FinSQL architecture is divided into three distinct modules designed to optimize the transition from natural language to valid SQL queries.
1. Prompt Construction
To mitigate data scarcity, FinSQL employs a hybrid augmentation strategy, generating Chain-of-Thought (CoT) reasoning paths and "SQL Skeletons" to enrich training data. A critical innovation here is the Parallel Cross-Encoder for schema linking. Unlike serial encoders that fail under the weight of hundreds of columns, this method processes tables in parallel batches, rapidly identifying relevant schema items to prune the context for the LLM.
2. Parameter-Efficient Fine-Tuning (PEFT)
FinSQL utilizes Low-Rank Adaptation (LoRA) to train specialized "plugins" for specific financial tasks. These adapters are stored in a LoRA Plugin Hub. When faced with a new database or a low-resource scenario, the framework can merge weights from existing plugins (e.g., combining "Stock" and "Fund" knowledge) to provide a "warm start," significantly boosting performance without requiring exhaustive retraining.
3. Output Calibration
Financial databases are often too large for execution-based error checking to be practical. FinSQL introduces a non-execution-based calibration step. It generates multiple candidate queries and clusters them based on structural consistency. A final alignment check ensures that every column referenced in the SELECT clause exists within the tables defined in the FROM clause, effectively reducing hallucinations.
Key Results and Insights
FinSQL demonstrates that specialized, open-source frameworks can exceed the performance of general-purpose closed-source APIs in niche domains:
- Performance: Using LLaMA2-13B, FinSQL achieved an execution accuracy (EX) of 82.2% on BULL-en, notably higher than the 75.0% achieved by GPT-4-based methods.
- Transferability: In 10-shot scenarios, the LoRA weight merging method improved performance by up to 36.64% compared to standard fine-tuning, proving its efficacy for cross-database generalization.
- Calibration Efficiency: The parallel schema linking achieved an AUC of over 0.99, validating its ability to filter "wide" databases accurately without losing critical context.
Conclusion
FinSQL provides a robust, privacy-preserving solution for financial Text-to-SQL tasks. By addressing the "wide table" problem and enabling efficient knowledge transfer through LoRA merging, it offers a scalable path for financial institutions to leverage LLMs. Future work will focus on closing the performance gap in multilingual contexts and enhancing the framework's ability to handle multi-step financial reasoning.
Summary updated