Can reasoning models comprehend mathematical problems in Chinese ancient texts? An empirical study based on data from Suanjing Shishu

Chang Liu, Dongbo Wang, Zhixiao ZhaoarXiv 2025 · DOI · arXiv

Introduction

Modern benchmarks for mathematical reasoning, such as GSM8K or MATH, predominantly focus on contemporary English or Mandarin. This focus creates a "cultural homogeneity" in AI evaluation, overlooking the rich mathematical heritage found in classical texts. Classical Chinese presents unique challenges: it is highly concise, lacks traditional punctuation, and embeds logic within historical frameworks like Yin-Yang theory or ancient astronomical systems.

Researchers have introduced Guji_MATH, the first benchmark designed to evaluate whether large language models (LLMs) can interpret and solve mathematical problems from the Suanjing Shishu (Ten Mathematical Classics), bridging the gap between humanities and STEM. Without this capability, we risk losing the traditional calculation logic and techniques buried in ancient texts that modern scholars often struggle to interpret.

Methodology

The researchers developed a comprehensive pipeline to transform raw ancient texts into a structured, machine-readable benchmark.

The Guji_MATH evaluation pipeline, covering data construction, model selection, and metrics calculation.

The process involves four key stages:

  1. Punctuation & Extraction: Utilizing the specialized Xunzi model series to add punctuation to unformatted texts from the Suanjing Shishu.
  2. Structuring: Organizing problems into "Question-Answer-Solution" triplets. A critical innovation is the manual identification of "hidden premises"—contextual information (like grain conversion ratios) mentioned in earlier chapters but omitted in subsequent problems.
  3. Annotation: Tagging 538 problems by difficulty (Levels 1–4) and mathematical category (15 types, including linear equations and Pythagorean theorems).
  4. Dual Evaluation Modes:
    • Closed-Book: Solving problems from scratch using only the question and premises.
  5. Open-Book: Providing the ancient solution method to test the model's ability to translate archaic algorithms into modern logic.

The Guji_MATH Dataset

The dataset comprises 538 problems extracted from 8 canonical texts, primarily the Suanjing Shishu. Unlike modern datasets, Guji_MATH requires models to handle significant linguistic and cultural nuances. While these models often score over 90% on modern benchmarks like Math-500, their performance drops drastically on Guji_MATH, highlighting the "linguistic tax" of classical Chinese.

Examples of structured problems from Guji_MATH, illustrating the concepts of hidden and shared premises.

As shown in Figure 3, "hidden premises" are essential for solving these problems but are often implicit in the source text. This adds a layer of complexity beyond pure mathematical calculation, requiring models to possess historical and cultural context.

Key Insights and Results

The study revealed several critical findings regarding the capabilities of modern LLMs:

  • The "Context Gap": Nearly 23% of model failures were attributed to a lack of historical context, such as misunderstanding ancient metrological standards or unit conversion rules.
  • Performance Leaders: DeepSeek R1 achieved the highest accuracy in Closed-Book mode (63.01%), while Qwen-Plus led in Open-Book mode (70.25%).
  • Scale and Reasoning: Trillion-parameter models significantly outperformed smaller 32B models, suggesting that scale is vital for navigating low-resource linguistic tasks.
  • The Open-Book Boost: Providing the ancient solution significantly improved performance for almost all models, suggesting that while LLMs have the reasoning power, they struggle with the initial comprehension of archaic phrasing.
  • Difficulty Scaling: Accuracy dropped sharply as problem complexity increased. For Level 4 problems, even top models struggled to exceed 50% accuracy in Closed-Book settings.

Conclusion

While top-tier models show promise, their performance on Guji_MATH is significantly lower than on modern benchmarks. Classical Chinese's conciseness, where subjects or operators are often omitted, remains a major barrier. Future work aims to expand the dataset to include the "Golden Age" of Chinese mathematics (Song, Yuan, and Ming dynasties) and explore standardization techniques like pseudo-code translation to bridge the comprehension gap.

Summary updated

Abstract

This study addresses the challenges in intelligent processing of Chinese ancient mathematical classics by constructing Guji_MATH, a benchmark for evaluating classical texts based on Suanjing Shishu. It systematically assesses the mathematical problem-solving capabilities of mainstream reasoning models under the unique linguistic constraints of classical Chinese. Through machine-assisted annotation and manual verification, 538 mathematical problems were extracted from 8 canonical texts, forming a structured dataset centered on the "Question-Answer-Solution" framework, supplemented by problem types and difficulty levels. Dual evaluation modes--closed-book (autonomous problem-solving) and open-book (reproducing classical solution methods)--were designed to evaluate the performance of six reasoning models on ancient Chinese mathematical problems. Results indicate that reasoning models can partially comprehend and solve these problems, yet their overall performance remains inferior to benchmarks on modern mathematical tasks. Enhancing models' classical Chinese comprehension and cultural knowledge should be prioritized for optimization. This study provides methodological support for mining mathematical knowledge from ancient texts and disseminating traditional culture, while offering new perspectives for evaluating cross-linguistic and cross-cultural capabilities of reasoning models.
BibTeX
@misc{liu2025can,
  author = {Chang Liu and Dongbo Wang and Liu Liu and Zhixiao Zhao},
  title = {{Can reasoning models comprehend mathematical problems in Chinese ancient texts? An empirical study based on data from Suanjing Shishu}},
  year = {2025},
  eprint = {2505.16660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.16660},
}
Figure