JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams

Issa Sugiura, Taro Makino, Takashi IshidaarXiv 2026 · arXiv

As Large Language Models (LLMs) move from general-purpose assistants to specialized professional tools, evaluating them in high-stakes domains like finance becomes critical. However, financial systems are deeply tied to local regulations, and static benchmarks quickly become "contaminated" as their questions leak into model training data.

JFinBench addresses these challenges by introducing a continuously refreshable benchmark based on the latest Japanese financial certification exams.

TL;DR

JFinBench is a new Japanese financial benchmark featuring 321 examples from 2026 certification exams, designed to combat data contamination through a "live" update pipeline. While proprietary models like Claude Fable 5 and GPT-6 Astra lead with scores over 93%, the study reveals that providing models with web search and terminal access significantly narrows the performance gap between open-weight and proprietary models.

What Problem Does This Paper Address?

The authors identify two major hurdles in evaluating LLMs for finance:

  1. Locality: Most financial benchmarks are U.S.-centric. A model that understands the U.S. tax code is useless for a Japanese accountant.
  2. Contamination: Once a benchmark is public, it's often scraped into the training sets of future models. This leads to "memorization" rather than true reasoning, inflating scores over time.

Existing Japanese benchmarks were either too small, limited to multiple-choice questions, or used outdated exam editions that models had likely already seen.

Methods

To build a scalable, refreshable benchmark, the authors developed a human-in-the-loop agentic pipeline. Converting complex exam PDFs (filled with tables, figures, and specific formatting) into structured data is notoriously difficult for standard tools.

The JFinBench construction pipeline uses VLMs for transcription and coding agents for structuring, with a final human verification step.

The pipeline works in three stages:

  • Generic Transcription: A Vision-Language Model (VLM) transcribes the PDF into layout-preserving Markdown.
  • Agentic Conversion: A coding agent (Claude Code) writes custom Python scripts for each exam to turn the transcripts into structured JSON records.
  • Human Verification: A native speaker reviews the output side-by-side with the original PDF, feeding natural language corrections back to the agent until the data is perfect.

Datasets

JFinBench consists of 321 examples drawn from the 2026 editions of six major Japanese certification exams:

  • CPA (short-answer)
  • Tax Accountant
  • Financial Planning (Level 1)
  • Banking Counter Services
  • Nissho & Zensho Bookkeeping

Unlike previous efforts, JFinBench includes free-response, form-filling, and image-dependent (VQA) questions.

Comparison of JFinBench with previous Japanese financial benchmarks.

Key Insights and Ideas

  • Knowledge is the Bottleneck: Failure analysis shows that "missing knowledge" is the primary reason models fail, far outweighing calculation errors or reasoning flaws. Distribution of error types across different models. Knowledge gaps are the dominant failure mode.

  • Tools are the Great Equalizer: When models are given access to a terminal and web search (using the Harbor framework), their scores jump significantly. Open-weight models like Kimi K3.1 Pro saw gains of nearly 10 points, narrowing the gap with top-tier proprietary models from 12% to just 4%.

  • Parametric Knowledge Still Matters: Even with tools, models that performed better "closed-book" tended to use tools more effectively. Internal knowledge helps the agent decide what to look up and how to apply the retrieved information.
Distribution of error types across different models. Knowledge gaps are the dominant failure mode.

Key Results

The evaluation covered 12 recent LLMs in two settings: "without tools" (parametric knowledge) and "with tools" (agentic search/compute).

  • Proprietary Dominance: Claude Fable 5 and GPT-6 Astra achieved the highest scores (95.8% and 93.6% respectively without tools).
  • Open-Weight Progress: Kimi K3.1 Pro emerged as the strongest open-weight model, though it still trailed the leaders significantly before tool use was introduced.
  • Scaling Trends: Performance generally correlated with parameter count, but post-training played a massive role—Kimi K3.1 Pro significantly outperformed larger models like Mistral Large 4.
Model performance vs. API cost, showing the trade-off between reasoning effort and accuracy.

Limitations and Future Work

  • Saturation: Top proprietary models are already nearing 100% on several exams. Future versions of JFinBench will need to filter for only the hardest questions or incorporate even more advanced professional certifications.
  • Coverage: The benchmark is limited to exams that publish their questions as PDFs. Some high-level exams (like the Securities Analyst exam) remain out of scope.
  • Contamination Risks: While using 2026 exams mitigates current contamination, the authors acknowledge that refreshing the benchmark annually is the only long-term solution.

By open-sourcing the construction pipeline alongside the data, the authors have provided a blueprint for keeping domain-specific benchmarks relevant in the rapidly evolving LLM landscape.

Summary updated

Abstract

Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
BibTeX
@misc{sugiura2026jfinbench,
  author = {Issa Sugiura and Taro Makino and Takashi Ishida},
  title = {{JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams}},
  year = {2026},
  eprint = {2611.18130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18130},
}
Figure