TL;DR
This paper introduces QuantEval, a new benchmark for evaluating Large Language Models (LLMs) on crucial quantitative finance tasks, including knowledge-based QA, mathematical reasoning, and strategy coding. By integrating a realistic backtesting framework, QuantEval reveals significant performance gaps between current LLMs and human experts, particularly in complex reasoning and coding, while also demonstrating the potential of domain-specific fine-tuning.
What Problem Does This Paper Address?
The rapid advancement of LLMs has spurred interest in their application to finance. However, existing financial benchmarks primarily focus on knowledge retrieval and simple question answering, failing to capture the multifaceted skills required in quantitative finance. Real-world quantitative finance involves not only understanding financial concepts but also performing precise multi-step calculations grounded in market data and translating these into executable trading strategies that can be rigorously evaluated through backtesting. This gap means current LLM evaluations don't accurately reflect their readiness for practical quantitative finance roles.
Methods
QuantEval addresses this by evaluating LLMs across three key dimensions:
- Knowledge-based QA: Assesses conceptual understanding of financial definitions, theories, and terminology.
- Quantitative Mathematical Reasoning: Tests the ability to solve multi-step quantitative problems using real market data and financial formulas.
- Quantitative Strategy Coding: Evaluates the generation of executable trading strategies within a standardized CTA-style backtesting framework. This is a novel aspect, as it moves beyond simple code compilation to assess actual strategy performance using financial metrics.
The benchmark construction involved a pipeline combining expert annotation and multi-agent generation, followed by rigorous validation. For strategy coding, a deterministic backtesting environment with specified asset universes, costs, and risk constraints is used to ensure reproducibility.
Datasets
QuantEval itself is a newly introduced benchmark comprising 1,575 carefully curated samples. These are divided into:
- Knowledge-based QA: 660 samples.
- Quantitative Mathematical Reasoning: 855 samples.
- Quantitative Strategy Coding: 60 samples.
The data for QuantEval is sourced from publicly available financial textbooks, academic papers, regulatory filings, curated glossaries, historical market data, macroeconomic indicators, and expert-validated trading strategy templates. The paper also mentions using existing financial datasets like FinQA, TAT-QA, FinEval, FinanceIQ, and Finova for comparative analysis.
Key Insights and Ideas
- LLMs struggle with quantitative finance: Despite impressive general capabilities, LLMs show substantial performance gaps compared to human experts, especially in quantitative reasoning and strategy coding.
- Backtesting is crucial for strategy evaluation: Simply checking code syntax is insufficient. Evaluating generated strategies within a realistic backtesting framework is essential to understand their financial viability and risk.
- Chain-of-Thought (CoT) prompting helps, but isn't a silver bullet: CoT significantly improves reasoning performance but can sometimes lead to overthinking or hallucinations for factual recall tasks.
- Strategy coding is the hardest dimension: Generating executable and financially sound trading strategies is a major bottleneck for current LLMs, with many open-source models failing to produce even basic executable code.
- Domain-specific training matters: Supervised fine-tuning (SFT) and reinforcement learning (RL) on domain-aligned data can lead to consistent improvements in LLM performance on quantitative finance tasks.
Key Results
- Human experts significantly outperform LLMs: Human experts achieve accuracies around 90% for QA and Reasoning, while the best LLMs lag behind, particularly in reasoning.
- Proprietary models lead, but gaps remain: Closed-source models generally outperform open-source models across all tasks, but still fall short of human expert performance.
- Strategy Coding Executability: Most open-source models have near-zero executable rates for strategy coding, while proprietary models show improved but still limited executability. For instance, GPT-5 achieved a 51.7% executable rate with CoT, compared to 8.3% for Qwen3-30B-A3B.
- Reasoning Accuracy: GPT-5 achieved 55.0% accuracy on reasoning with CoT, while human experts reached 89.05%.
- Domain-aligned training shows promise: Extended training with SFT and GRPO RL on DianJin-R1-7B improved its reasoning performance, though the gap to humans remained.
Limitations and Future Work
- English-only focus: The current benchmark is limited to English instances, which may not fully capture the capabilities needed for multilingual financial markets.
- Fixed backtesting assumptions: The strategy coding task uses a fixed CTA-style backtesting framework. Different configurations might yield different results.
- Limited strategy coding samples: The 60 strategy coding instances are fewer than other categories due to the high cost of expert validation. Scaling this aspect is a future direction.
- Future work: The authors suggest expanding the benchmark to include multilingual instances, exploring diverse backtesting configurations, and increasing the number of strategy coding tasks. They also highlight the need for further research into training strategies that can effectively bridge the gap to human expert performance.
Summary updated