FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging

Zichen Tang, Haihong E, Jiacheng Liu, Zhongjun Yang, Rongjin Li, Zihua Rong, Haoyang He, Zhuodi Hao, Xinyang Hu, Kun Ji, Ziyan Ma, Mengyuan Ji, Jun Zhang, Chenghao Ma, Qianhe Zheng, Yang Liu, Yiling Huang, Xinyi Hu, Qing Huang, Zijian Xie, Shiyao Peng

Abstract

We present FinMMR, a novel bilingual multimodal benchmark tailored to evaluate the reasoning capabilities of multimodal large language models (MLLMs) in financial numerical reasoning tasks. Compared to existing benchmarks, our work introduces three significant advancements: multimodality across 14 image categories, comprehensiveness spanning 14 financial subdomains, and high-level challenge requiring multi-step precise numerical reasoning. The best-performing MLLM achieves only 53.0% accuracy on "Hard" problems, highlighting a significant gap in current model capabilities for real-world financial scenarios.

Introduction

Financial analysis is a high-stakes field that demands more than just textual comprehension; it requires synthesizing information from dense tables, intricate charts, and complex diagrams to perform precise calculations. While Large Language Models (LLMs) have become proficient at text-based mathematics, their multimodal counterparts (MLLMs) often falter when financial data is presented visually. FinMMR is a new, rigorous bilingual benchmark designed to push the limits of MLLMs by combining complex visual data—like ownership structure charts and candlestick graphs—with multi-step numerical reasoning.

Problem Statement

Existing benchmarks in the financial AI space are often too narrow, rely on multiple-choice formats that allow for lucky guesses, or provide data as structured text rather than raw images. This fails to capture the complexity of professional financial analysis. FinMMR addresses these gaps by creating a dataset that forces models to perceive, reason, and calculate directly from diverse financial visuals across 14 distinct subdomains, ranging from corporate finance to derivatives.

Methods

The researchers introduced several novel evaluative and architectural approaches to improve model performance:

  • Program-of-Thought (PoT) vs. CoT: Beyond standard step-by-step reasoning (Chain-of-Thought), the authors emphasize PoT, where models generate executable Python code to solve problems. This reduces calculation errors and improves token efficiency.
  • Visual Filtering Pipeline: To mimic real-world noise, they mixed "ground truth" images with "distractor" images. They proposed a two-stage pipeline where the model first filters for relevance before attempting the reasoning task.
  • Knowledge Augmentation (RAG): They built a library of over 3,100 financial Python functions. Models were tested on their ability to retrieve and apply these specialized functions to solve domain-specific problems.
  • Model Collaboration: The authors tested a "divide and conquer" strategy, using a strong vision model (like GPT-4o) to parse images into structured text, which is then fed into a specialized reasoning model (like DeepSeek-R1) to perform the math.
A complex financial reasoning example requiring visual parsing of an ownership structure and numerical calculation via Python code (PoT).

The FinMMR Dataset

The FinMMR benchmark is characterized by its breadth and depth:

  • Scale: 4,300 questions and 8,738 images in both English and Chinese.
  • Diversity: 14 image categories (including bar charts, line charts, ownership structures, and doughnut charts) and 14 subdomains (covering Banking, Macroeconomics, Insurance, and more).
  • Difficulty Tiers: Problems are algorithmically categorized into Easy, Medium, and Hard based on the number of operators and lines of code required to solve them.
  • Real-World Data: Includes CRRQA, a subset derived from 90 recent Chinese financial research reports to ensure the data reflects "in the wild" professional standards.
Overview of FinMMR's core pillars: rich imagery, comprehensive subdomains, and complex formulas.

Key Insights and Results

The "Visual Distractor" Tax

Simply adding irrelevant images to a prompt caused a significant performance drop (over 10% for some models). This suggests that current MLLMs lack robust "visual attention" and are easily distracted by noise.

Code-Based Reasoning

Program-of-Thought (PoT) generally outperformed text-based reasoning. By offloading arithmetic to a Python interpreter, models avoided the "hallucinated math" common in standard LLM outputs. Furthermore, providing a library of financial functions (Knowledge Augmentation) boosted accuracy by up to 4.3 percentage points.

Performance Benchmarks

  • The Hard Ceiling: On the "Hard" subset, Claude 3.7 Sonnet (with 64K thinking tokens) led the field but achieved only 53.0% accuracy. OpenAI o1 followed at 48.4%.
  • Thinking Costs: Models with "extended thinking" modes showed consistent improvements (about 2.2% for Claude 3.7) but at a massive cost—using nearly 12 times more tokens.
  • Error Distribution: Analysis showed a near-even split between Visual Perception Errors (30%), Knowledge Reasoning Errors (38%), and Numerical Computation Errors (32%), proving that the bottleneck is systemic across all stages of the pipeline.

Conclusion

FinMMR reveals that even top-tier models struggle with the integration of visual perception and complex financial logic. While "extended thinking" and model collaboration (Visual Parser + Reasoner) show promise, the high token costs and persistent errors in visual-textual alignment remain significant hurdles. Future work will focus on developing more token-efficient reasoning paths and improving the precision of visual parsing for multi-page financial documents.

Summary updated

Abstract

We present FinMMR, a novel bilingual multimodal benchmark tailored to evaluate the reasoning capabilities of multimodal large language models (MLLMs) in financial numerical reasoning tasks. Compared to existing benchmarks, our work introduces three significant advancements. (1) Multimodality: We meticulously transform existing financial reasoning benchmarks, and construct novel questions from the latest Chinese financial research reports. FinMMR comprises 4.3K questions and 8.7K images spanning 14 categories, including tables, bar charts, and ownership structure charts. (2) Comprehensiveness: FinMMR encompasses 14 financial subdomains, including corporate finance, banking, and industry analysis, significantly exceeding existing benchmarks in financial domain knowledge breadth. (3) Challenge: Models are required to perform multi-step precise numerical reasoning by integrating financial knowledge with the understanding of complex financial images and text. The best-performing MLLM achieves only 53.0% accuracy on Hard problems. We believe that FinMMR will drive advancements in enhancing the reasoning capabilities of MLLMs in real-world scenarios.
BibTeX
@inproceedings{tang2025finmmr,
  author = {Zichen Tang and Haihong E and Jiacheng Liu and Zhongjun Yang and Rongjin Li and Zihua Rong and Haoyang He and Zhuodi Hao and Xinyang Hu and Kun Ji and Ziyan Ma and Mengyuan Ji and Jun Zhang and Chenghao Ma and Qianhe Zheng and Yang Liu and Yiling Huang and Xinyi Hu and Qing Huang and Zijian Xie and Shiyao Peng},
  title = {{FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure