FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning

Liwen ZhangarXiv 2025 · DOI · arXiv

Abstract

Evaluating Large Language Models (LLMs) in specialized domains like finance is challenging due to the entanglement of domain knowledge and reasoning logic. FinEval-KR introduces a novel evaluation framework that decouples these two capabilities, allowing for precise error diagnosis. By utilizing a three-stage pipeline and a massive Chinese financial dataset annotated with Bloom’s Taxonomy, the framework identifies whether model failures stem from knowledge gaps or reasoning deficiencies.

Introduction

Current benchmarks for LLMs in finance often provide a single accuracy score, failing to distinguish between a model's lack of factual knowledge and its inability to perform complex, multi-step reasoning. This "entanglement" prevents developers from performing effective root-cause analysis. FinEval-KR addresses this by moving beyond simple pass/fail metrics to provide a diagnostic "medical report" for LLM performance, specifically focusing on the bottleneck of applying facts to high-order cognitive tasks.

Methodology: The Three-Stage Pipeline

The core of FinEval-KR is a three-stage evaluation process designed to isolate variables and pinpoint the source of errors.

The three-stage evaluation pipeline of FinEval-KR, illustrating the transition from standard QA to knowledge-augmented answering and final error diagnosis.
  1. Stage 1: Question Answering: The model attempts a financial problem. An "LLM-as-a-judge" identifies errors and extracts the specific "knowledge points" (e.g., formulas or facts) the model failed to utilize.
  2. Stage 2: Knowledge-Augmented Answering: The framework provides the model with the missing knowledge identified in Stage 1 and asks it to re-attempt the problem.
  3. Stage 3: Error Diagnosis: By comparing the two attempts, the framework attributes the failure. If the model succeeds in Stage 2, the original error is classified as a Knowledge Deficiency. If it still fails, it is a Reasoning Ability Deficiency.

Furthermore, the framework incorporates Bloom’s Taxonomy, mapping reasoning steps to five cognitive levels: Remembering, Understanding, Applying, Analyzing, and Evaluating. This allows for the calculation of a Cognitive Score (CS) to identify specific cognitive bottlenecks.

Dataset and Annotation

The researchers released a new open-source Chinese financial reasoning dataset containing 9,782 question-answer pairs. The source material is derived from nine canonical financial textbooks covering 22 subfields, such as Corporate Finance and Monetary Policy. Unlike standard datasets, each sample includes specific knowledge point labels, step-by-step reasoning chains, and the required cognitive level for each step. Quality control involved a triple-validation process with OpenAI's o1 model and 30 human experts.

Results and Discussion

Evaluation of 18 leading LLMs revealed several key insights:

  • Knowledge vs. Reasoning: Top-tier models like DeepSeek-R1 and Gemini-2.5-pro demonstrate high Knowledge Scores (KS > 0.90), suggesting that factual knowledge is largely a solved problem. However, the Reasoning Score (RS) remains the primary differentiator; DeepSeek-R1 led with an RS of 0.9347, while older models like GPT-3.5-turbo languished at 0.3973.
  • The "Application" Bottleneck: Models frequently struggle with the "Applying" (CS3) category of Bloom's Taxonomy. While they can discuss complex theories, they often fail to apply them to specific numerical scenarios.
  • Generalist Superiority: General foundation models consistently outperform specialized financial LLMs by over 20%. For instance, Xuanyuan-FinX1-preview achieved an accuracy of 0.5890, significantly lower than the 0.8750 achieved by Gemini-2.5-pro.
An example of a diagnostic review result generated by the framework, identifying the cause of error, location, and missing knowledge.

The diagnostic output, as shown in the figure above, provides granular feedback on why a model failed, such as failing to convert an annual interest rate to a monthly one despite knowing the general concept.

Conclusion

FinEval-KR provides a sophisticated diagnostic tool for the financial AI community. By decoupling knowledge from reasoning, it reveals that the true bottleneck for AI in finance is the application of facts to multi-step logic. Future work will focus on multi-judge pipelines to reduce bias and the inclusion of multi-modal financial data.

Summary updated

BibTeX
@misc{zhang2025fineval,
  author = {Liwen Zhang},
  title = {{FinEval-KR: A Financial Domain Evaluation Framework for Large Language Models' Knowledge and Reasoning}},
  year = {2025},
  eprint = {2506.21591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.21591},
}
Figure