On Path to Multimodal Historical Reasoning: HistBench and HistAgent

Jiahao Qiu, Fulian Xiao, Yimin Wang, Yuchen Mao, Yijia Chen, Xinzhe Juan, Siran Wang, Xuan Qi, Tongcheng Zhang, Zixin Yao, Jiacheng Guo, Yifu Lu, Charles Argon, Jundi Cui, Daixin Chen, Junran Zhou, Shuyao Zhou, Zhanpeng Zhou, Ling Yang, Shilong Liu, Hongru Wang, Kaixuan Huang, Xun Jiang, Yuming Cao, Yue Chen, Yunfei Chen, Zhengyi Chen, Ruowei Dai, Mengqiu Deng, Jiye Fu, Yunting Gu, Zijie Guan, Zirui Huang, Xiaoyan Ji, Yumeng Jiang, Delong Kong, Haolong Li, Jiaqi Li, Ruipeng Li, Tianze Li, Zhuoran Li, Haixia Lian, Mengyue Lin, Xudong Liu, Jiayi Lu, Jinghan Lu, Wanyu Luo, Ziyue Luo, Zihao Pu, Zhi Qiao, Ruihuan Ren, Liang Wan, Ruixiang Wang, Tianhui Wang, Yang Wang, Zeyu Wang, Zihua Wang, Yujia Wu, Zhaoyi Wu, Hao Xin, Weiao Xing, Ruojun Xiong, Weijie Xu, Yao Shu, Xiao Yao, Xiaorui Yang, Yuchen Yang, Nan Yi, Jiadong Yu, Yangyuxuan Yu, Huiting Zeng, Danni Zhang, Yunjie Zhang, Zhaoyu Zhang, Zhiheng Zhang, Xiaofeng Zheng, Peirong Zhou, Linyan Zhong, Xiaoyin Zong, Ying Zhao, Zhenxin Chen, Lin Ding, Xiaoyu Gao, Bingbing Gong, Yichao Li, Yang Liao, Guang Ma, Tianyuan Ma, Xinrui Sun, Tianyi Wang, Han Xia, Ruobing Xian, Gen Ye, Tengfei Yu, Wentao Zhang, Yuxi Wang, Mengdi Wang

Introduction

While Artificial Intelligence has made significant strides in scientific and legal domains, historical reasoning remains a "final frontier." Historical inquiry is not merely the recitation of dates; it requires "epistemic grounding"—the sophisticated ability to interpret fragmented, multimodal primary sources such as 18th-century handwritten deeds, weathered inscriptions, or obscure cartographic materials. Current Large Language Models (LLMs) often fail in this arena due to perceptual failures (e.g., inability to read cursive), linguistic barriers (struggling with "dead" or low-resource languages), and a lack of methodological rigor inherent to professional historical workflows.

HistAgent: A Specialized AI Historian

To bridge these gaps, the authors introduce HistAgent, an agentic system designed with a "Manager-Specialist" architecture. Built using the smolagents library, the Manager Agent utilizes a CodeAct-style loop to decompose complex historical queries into executable Python sub-tasks.

Unlike general-purpose chatbots, HistAgent is equipped with a suite of specialized toolkits:

  • OCR Agent: Integrates Transkribus for Western manuscripts and custom models for Asian scripts.
  • Literature Search Agent: Employs a multi-stage protocol prioritizing scholarly databases like Google Books and Springer.
  • Image Information Agent: Uses reverse image search to establish the provenance of visual artifacts.
  • Translator Agent: Supports 29 languages, ensuring historical nuances are preserved during translation.

The system concludes its process by generating a comprehensive "Summary Report" that details every tool utilized, citations for every source, and the step-by-step logic behind its conclusion.

The HistBench Benchmark

The study also presents HistBench, the first comprehensive benchmark specifically designed for historical reasoning. Authored by over 40 experts, including professors and PhDs, the benchmark consists of 414 high-quality questions. It spans 29 languages and 20+ geographic regions across five major historical epochs.

A key feature of HistBench is its difficulty stratification, which categorizes tasks from Level 1 (basic factual retrieval) to Level 3 (challenging interdisciplinary synthesis) across multiple dimensions of complexity.

Difficulty stratification levels in HistBench across six dimensions of historical reasoning.

Key Results and Insights

Experimental results demonstrate that domain-specific tools are more critical than model scale alone. HistAgent, powered by GPT-4o, significantly outperformed much larger reasoning models and generalist search agents.

Performance comparison on HistBench across various models and agent configurations.

On HistBench, HistAgent achieved a 27.54% pass@1 and 36.47% pass@2 accuracy. In comparison, GPT-4o with standard online search reached only 18.60%, and DeepSeek-R1 trailed at 14.49%. Furthermore, on the history subset of Humanity’s Last Exam (HLE), HistAgent more than tripled the performance of standard GPT-4o (28.57% vs 8.93%).

Crucially, the "specialization tax" was non-existent; HistAgent maintained high generalist capabilities, achieving 60% accuracy on the GAIA benchmark. This suggests that equipping agents with specialized historical tools enhances their reasoning without degrading general performance.

Conclusion and Future Directions

The primary bottleneck for AI in the humanities remains the "Cursive Wall"—the difficulty of digitizing and interpreting primary source data. By integrating specialized OCR and archival search, HistAgent begins to dismantle this barrier. While current evaluations are limited to closed-ended questions for objectivity, future work aims to explore open-ended historical debates and counterfactual reasoning using even more advanced foundation models.

Summary updated

BibTeX
@inproceedings{qiu2026path,
  author = {Jiahao Qiu and Fulian Xiao and Yimin Wang and Yuchen Mao and Yijia Chen and Xinzhe Juan and Siran Wang and Xuan Qi and Tongcheng Zhang and Zixin Yao and Jiacheng Guo and Yifu Lu and Charles Argon and Jundi Cui and Daixin Chen and Junran Zhou and Shuyao Zhou and Zhanpeng Zhou and Ling Yang and Shilong Liu and Hongru Wang and Kaixuan Huang and Xun Jiang and Yuming Cao and Yue Chen and Yunfei Chen and Zhengyi Chen and Ruowei Dai and Mengqiu Deng and Jiye Fu and Yunting Gu and Zijie Guan and Zirui Huang and Xiaoyan Ji and Yumeng Jiang and Delong Kong and Haolong Li and Jiaqi Li and Ruipeng Li and Tianze Li and Zhuoran Li and Haixia Lian and Mengyue Lin and Xudong Liu and Jiayi Lu and Jinghan Lu and Wanyu Luo and Ziyue Luo and Zihao Pu and Zhi Qiao and Ruihuan Ren and Liang Wan and Ruixiang Wang and Tianhui Wang and Yang Wang and Zeyu Wang and Zihua Wang and Yujia Wu and Zhaoyi Wu and Hao Xin and Weiao Xing and Ruojun Xiong and Weijie Xu and Yao Shu and Xiao Yao and Xiaorui Yang and Yuchen Yang and Nan Yi and Jiadong Yu and Yangyuxuan Yu and Huiting Zeng and Danni Zhang and Yunjie Zhang and Zhaoyu Zhang and Zhiheng Zhang and Xiaofeng Zheng and Peirong Zhou and Linyan Zhong and Xiaoyin Zong and Ying Zhao and Zhenxin Chen and Lin Ding and Xiaoyu Gao and Bingbing Gong and Yichao Li and Yang Liao and Guang Ma and Tianyuan Ma and Xinrui Sun and Tianyi Wang and Han Xia and Ruobing Xian and Gen Ye and Tengfei Yu and Wentao Zhang and Yuxi Wang and Xi Gao and Mengdi Wang},
  title = {{On Path to Multimodal Historical Reasoning: HistBench and HistAgent}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=hh1FITGR8C},
}
Figure