Abstract
We present FinMMR, a novel bilingual multimodal benchmark tailored to evaluate the reasoning capabilities of multimodal large language models (MLLMs) in financial numerical reasoning tasks. Compared to existing benchmarks, our work introduces three significant advancements: multimodality across 14 image categories, comprehensiveness spanning 14 financial subdomains, and high-level challenge requiring multi-step precise numerical reasoning. The best-performing MLLM achieves only 53.0% accuracy on "Hard" problems, highlighting a significant gap in current model capabilities for real-world financial scenarios.
Introduction
Financial analysis is a high-stakes field that demands more than just textual comprehension; it requires synthesizing information from dense tables, intricate charts, and complex diagrams to perform precise calculations. While Large Language Models (LLMs) have become proficient at text-based mathematics, their multimodal counterparts (MLLMs) often falter when financial data is presented visually. FinMMR is a new, rigorous bilingual benchmark designed to push the limits of MLLMs by combining complex visual data—like ownership structure charts and candlestick graphs—with multi-step numerical reasoning.
Problem Statement
Existing benchmarks in the financial AI space are often too narrow, rely on multiple-choice formats that allow for lucky guesses, or provide data as structured text rather than raw images. This fails to capture the complexity of professional financial analysis. FinMMR addresses these gaps by creating a dataset that forces models to perceive, reason, and calculate directly from diverse financial visuals across 14 distinct subdomains, ranging from corporate finance to derivatives.
Methods
The researchers introduced several novel evaluative and architectural approaches to improve model performance:
- Program-of-Thought (PoT) vs. CoT: Beyond standard step-by-step reasoning (Chain-of-Thought), the authors emphasize PoT, where models generate executable Python code to solve problems. This reduces calculation errors and improves token efficiency.
- Visual Filtering Pipeline: To mimic real-world noise, they mixed "ground truth" images with "distractor" images. They proposed a two-stage pipeline where the model first filters for relevance before attempting the reasoning task.
- Knowledge Augmentation (RAG): They built a library of over 3,100 financial Python functions. Models were tested on their ability to retrieve and apply these specialized functions to solve domain-specific problems.
- Model Collaboration: The authors tested a "divide and conquer" strategy, using a strong vision model (like GPT-4o) to parse images into structured text, which is then fed into a specialized reasoning model (like DeepSeek-R1) to perform the math.
The FinMMR Dataset
The FinMMR benchmark is characterized by its breadth and depth:
- Scale: 4,300 questions and 8,738 images in both English and Chinese.
- Diversity: 14 image categories (including bar charts, line charts, ownership structures, and doughnut charts) and 14 subdomains (covering Banking, Macroeconomics, Insurance, and more).
- Difficulty Tiers: Problems are algorithmically categorized into Easy, Medium, and Hard based on the number of operators and lines of code required to solve them.
- Real-World Data: Includes CRRQA, a subset derived from 90 recent Chinese financial research reports to ensure the data reflects "in the wild" professional standards.
Key Insights and Results
The "Visual Distractor" Tax
Simply adding irrelevant images to a prompt caused a significant performance drop (over 10% for some models). This suggests that current MLLMs lack robust "visual attention" and are easily distracted by noise.
Code-Based Reasoning
Program-of-Thought (PoT) generally outperformed text-based reasoning. By offloading arithmetic to a Python interpreter, models avoided the "hallucinated math" common in standard LLM outputs. Furthermore, providing a library of financial functions (Knowledge Augmentation) boosted accuracy by up to 4.3 percentage points.
Performance Benchmarks
- The Hard Ceiling: On the "Hard" subset, Claude 3.7 Sonnet (with 64K thinking tokens) led the field but achieved only 53.0% accuracy. OpenAI o1 followed at 48.4%.
- Thinking Costs: Models with "extended thinking" modes showed consistent improvements (about 2.2% for Claude 3.7) but at a massive cost—using nearly 12 times more tokens.
- Error Distribution: Analysis showed a near-even split between Visual Perception Errors (30%), Knowledge Reasoning Errors (38%), and Numerical Computation Errors (32%), proving that the bottleneck is systemic across all stages of the pipeline.
Conclusion
FinMMR reveals that even top-tier models struggle with the integration of visual perception and complex financial logic. While "extended thinking" and model collaboration (Visual Parser + Reasoner) show promise, the high token costs and persistent errors in visual-textual alignment remain significant hurdles. Future work will focus on developing more token-efficient reasoning paths and improving the precision of visual parsing for multi-page financial documents.
Summary updated