QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

Zhaolu Kang, Junhao Gong, Wenqing Hu, Shuo Yin, Kehan Jiang, Zhicheng Fang, Yingjie He, Chunlei Meng, Rong Fu, Dongyang Chen, Leqi Zheng, Eric Hanchen Jiang, Yunfei Feng, Yitong Leng, Junfan Zhu, Xiaoyou Chen, Xi Yang, Richeng XuanarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces QuantEval, a new benchmark for evaluating Large Language Models (LLMs) on crucial quantitative finance tasks, including knowledge-based QA, mathematical reasoning, and strategy coding. By integrating a realistic backtesting framework, QuantEval reveals significant performance gaps between current LLMs and human experts, particularly in complex reasoning and coding, while also demonstrating the potential of domain-specific fine-tuning.

What Problem Does This Paper Address?

The rapid advancement of LLMs has spurred interest in their application to finance. However, existing financial benchmarks primarily focus on knowledge retrieval and simple question answering, failing to capture the multifaceted skills required in quantitative finance. Real-world quantitative finance involves not only understanding financial concepts but also performing precise multi-step calculations grounded in market data and translating these into executable trading strategies that can be rigorously evaluated through backtesting. This gap means current LLM evaluations don't accurately reflect their readiness for practical quantitative finance roles.

Methods

QuantEval addresses this by evaluating LLMs across three key dimensions:

  1. Knowledge-based QA: Assesses conceptual understanding of financial definitions, theories, and terminology.
  2. Quantitative Mathematical Reasoning: Tests the ability to solve multi-step quantitative problems using real market data and financial formulas.
  3. Quantitative Strategy Coding: Evaluates the generation of executable trading strategies within a standardized CTA-style backtesting framework. This is a novel aspect, as it moves beyond simple code compilation to assess actual strategy performance using financial metrics.

The benchmark construction involved a pipeline combining expert annotation and multi-agent generation, followed by rigorous validation. For strategy coding, a deterministic backtesting environment with specified asset universes, costs, and risk constraints is used to ensure reproducibility.

Datasets

QuantEval itself is a newly introduced benchmark comprising 1,575 carefully curated samples. These are divided into:

  • Knowledge-based QA: 660 samples.
  • Quantitative Mathematical Reasoning: 855 samples.
  • Quantitative Strategy Coding: 60 samples.

The data for QuantEval is sourced from publicly available financial textbooks, academic papers, regulatory filings, curated glossaries, historical market data, macroeconomic indicators, and expert-validated trading strategy templates. The paper also mentions using existing financial datasets like FinQA, TAT-QA, FinEval, FinanceIQ, and Finova for comparative analysis.

Key Insights and Ideas

  • LLMs struggle with quantitative finance: Despite impressive general capabilities, LLMs show substantial performance gaps compared to human experts, especially in quantitative reasoning and strategy coding.
  • Backtesting is crucial for strategy evaluation: Simply checking code syntax is insufficient. Evaluating generated strategies within a realistic backtesting framework is essential to understand their financial viability and risk.
  • Chain-of-Thought (CoT) prompting helps, but isn't a silver bullet: CoT significantly improves reasoning performance but can sometimes lead to overthinking or hallucinations for factual recall tasks.
  • Strategy coding is the hardest dimension: Generating executable and financially sound trading strategies is a major bottleneck for current LLMs, with many open-source models failing to produce even basic executable code.
  • Domain-specific training matters: Supervised fine-tuning (SFT) and reinforcement learning (RL) on domain-aligned data can lead to consistent improvements in LLM performance on quantitative finance tasks.

Key Results

  • Human experts significantly outperform LLMs: Human experts achieve accuracies around 90% for QA and Reasoning, while the best LLMs lag behind, particularly in reasoning.
  • Proprietary models lead, but gaps remain: Closed-source models generally outperform open-source models across all tasks, but still fall short of human expert performance.
  • Strategy Coding Executability: Most open-source models have near-zero executable rates for strategy coding, while proprietary models show improved but still limited executability. For instance, GPT-5 achieved a 51.7% executable rate with CoT, compared to 8.3% for Qwen3-30B-A3B.
  • Reasoning Accuracy: GPT-5 achieved 55.0% accuracy on reasoning with CoT, while human experts reached 89.05%.
  • Domain-aligned training shows promise: Extended training with SFT and GRPO RL on DianJin-R1-7B improved its reasoning performance, though the gap to humans remained.

Limitations and Future Work

  • English-only focus: The current benchmark is limited to English instances, which may not fully capture the capabilities needed for multilingual financial markets.
  • Fixed backtesting assumptions: The strategy coding task uses a fixed CTA-style backtesting framework. Different configurations might yield different results.
  • Limited strategy coding samples: The 60 strategy coding instances are fewer than other categories due to the high cost of expert validation. Scaling this aspect is a future direction.
  • Future work: The authors suggest expanding the benchmark to include multilingual instances, exploring diverse backtesting configurations, and increasing the number of strategy coding tasks. They also highlight the need for further research into training strategies that can effectively bridge the gap to human expert performance.

Summary updated

Abstract

Large Language Models (LLMs) have shown strong capabilities across many domains, yet their evaluation in financial quantitative tasks remains fragmented and mostly limited to knowledge-centric question answering. We introduce QuantEval, a benchmark that evaluates LLMs across three essential dimensions of quantitative finance: knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding. Unlike prior financial benchmarks, QuantEval integrates a CTA-style backtesting framework that executes model-generated strategies and evaluates them using financial performance metrics, enabling a more realistic assessment of quantitative coding ability. We evaluate some state-of-the-art open-source and proprietary LLMs and observe substantial gaps to human experts, particularly in reasoning and strategy coding. Finally, we conduct large-scale supervised fine-tuning and reinforcement learning experiments on domain-aligned data, demonstrating consistent improvements. We hope QuantEval will facilitate research on LLMs' quantitative finance capabilities and accelerate their practical adoption in real-world trading workflows. We additionally release the full deterministic backtesting configuration (asset universe, cost model, and metric definitions) to ensure strict reproducibility.
BibTeX
@misc{kang2026quanteval,
  author = {Zhaolu Kang and Junhao Gong and Wenqing Hu and Shuo Yin and Kehan Jiang and Zhicheng Fang and Yingjie He and Chunlei Meng and Rong Fu and Dongyang Chen and Leqi Zheng and Eric Hanchen Jiang and Yunfei Feng and Yitong Leng and Junfan Zhu and Xiaoyou Chen and Xi Yang and Richeng Xuan},
  title = {{QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models}},
  year = {2026},
  eprint = {2601.08689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08689},
}
Figure