As Large Language Models (LLMs) move from general-purpose assistants to specialized professional tools, evaluating them in high-stakes domains like finance becomes critical. However, financial systems are deeply tied to local regulations, and static benchmarks quickly become "contaminated" as their questions leak into model training data.
JFinBench addresses these challenges by introducing a continuously refreshable benchmark based on the latest Japanese financial certification exams.
TL;DR
JFinBench is a new Japanese financial benchmark featuring 321 examples from 2026 certification exams, designed to combat data contamination through a "live" update pipeline. While proprietary models like Claude Fable 5 and GPT-6 Astra lead with scores over 93%, the study reveals that providing models with web search and terminal access significantly narrows the performance gap between open-weight and proprietary models.
What Problem Does This Paper Address?
The authors identify two major hurdles in evaluating LLMs for finance:
- Locality: Most financial benchmarks are U.S.-centric. A model that understands the U.S. tax code is useless for a Japanese accountant.
- Contamination: Once a benchmark is public, it's often scraped into the training sets of future models. This leads to "memorization" rather than true reasoning, inflating scores over time.
Existing Japanese benchmarks were either too small, limited to multiple-choice questions, or used outdated exam editions that models had likely already seen.
Methods
To build a scalable, refreshable benchmark, the authors developed a human-in-the-loop agentic pipeline. Converting complex exam PDFs (filled with tables, figures, and specific formatting) into structured data is notoriously difficult for standard tools.
The pipeline works in three stages:
- Generic Transcription: A Vision-Language Model (VLM) transcribes the PDF into layout-preserving Markdown.
- Agentic Conversion: A coding agent (Claude Code) writes custom Python scripts for each exam to turn the transcripts into structured JSON records.
- Human Verification: A native speaker reviews the output side-by-side with the original PDF, feeding natural language corrections back to the agent until the data is perfect.
Datasets
JFinBench consists of 321 examples drawn from the 2026 editions of six major Japanese certification exams:
- CPA (short-answer)
- Tax Accountant
- Financial Planning (Level 1)
- Banking Counter Services
- Nissho & Zensho Bookkeeping
Unlike previous efforts, JFinBench includes free-response, form-filling, and image-dependent (VQA) questions.
Key Insights and Ideas
-
Knowledge is the Bottleneck: Failure analysis shows that "missing knowledge" is the primary reason models fail, far outweighing calculation errors or reasoning flaws. Distribution of error types across different models. Knowledge gaps are the dominant failure mode.
-
Tools are the Great Equalizer: When models are given access to a terminal and web search (using the Harbor framework), their scores jump significantly. Open-weight models like Kimi K3.1 Pro saw gains of nearly 10 points, narrowing the gap with top-tier proprietary models from 12% to just 4%.
- Parametric Knowledge Still Matters: Even with tools, models that performed better "closed-book" tended to use tools more effectively. Internal knowledge helps the agent decide what to look up and how to apply the retrieved information.
Key Results
The evaluation covered 12 recent LLMs in two settings: "without tools" (parametric knowledge) and "with tools" (agentic search/compute).
- Proprietary Dominance: Claude Fable 5 and GPT-6 Astra achieved the highest scores (95.8% and 93.6% respectively without tools).
- Open-Weight Progress: Kimi K3.1 Pro emerged as the strongest open-weight model, though it still trailed the leaders significantly before tool use was introduced.
- Scaling Trends: Performance generally correlated with parameter count, but post-training played a massive role—Kimi K3.1 Pro significantly outperformed larger models like Mistral Large 4.
Limitations and Future Work
- Saturation: Top proprietary models are already nearing 100% on several exams. Future versions of JFinBench will need to filter for only the hardest questions or incorporate even more advanced professional certifications.
- Coverage: The benchmark is limited to exams that publish their questions as PDFs. Some high-level exams (like the Securities Analyst exam) remain out of scope.
- Contamination Risks: While using 2026 exams mitigates current contamination, the authors acknowledge that refreshing the benchmark annually is the only long-term solution.
By open-sourcing the construction pipeline alongside the data, the authors have provided a blueprint for keeping domain-specific benchmarks relevant in the rapidly evolving LLM landscape.
Summary updated