Introduction
Modern benchmarks for mathematical reasoning, such as GSM8K or MATH, predominantly focus on contemporary English or Mandarin. This focus creates a "cultural homogeneity" in AI evaluation, overlooking the rich mathematical heritage found in classical texts. Classical Chinese presents unique challenges: it is highly concise, lacks traditional punctuation, and embeds logic within historical frameworks like Yin-Yang theory or ancient astronomical systems.
Researchers have introduced Guji_MATH, the first benchmark designed to evaluate whether large language models (LLMs) can interpret and solve mathematical problems from the Suanjing Shishu (Ten Mathematical Classics), bridging the gap between humanities and STEM. Without this capability, we risk losing the traditional calculation logic and techniques buried in ancient texts that modern scholars often struggle to interpret.
Methodology
The researchers developed a comprehensive pipeline to transform raw ancient texts into a structured, machine-readable benchmark.
The process involves four key stages:
- Punctuation & Extraction: Utilizing the specialized
Xunzimodel series to add punctuation to unformatted texts from the Suanjing Shishu. - Structuring: Organizing problems into "Question-Answer-Solution" triplets. A critical innovation is the manual identification of "hidden premises"—contextual information (like grain conversion ratios) mentioned in earlier chapters but omitted in subsequent problems.
- Annotation: Tagging 538 problems by difficulty (Levels 1–4) and mathematical category (15 types, including linear equations and Pythagorean theorems).
- Dual Evaluation Modes:
- Closed-Book: Solving problems from scratch using only the question and premises.
- Open-Book: Providing the ancient solution method to test the model's ability to translate archaic algorithms into modern logic.
The Guji_MATH Dataset
The dataset comprises 538 problems extracted from 8 canonical texts, primarily the Suanjing Shishu. Unlike modern datasets, Guji_MATH requires models to handle significant linguistic and cultural nuances. While these models often score over 90% on modern benchmarks like Math-500, their performance drops drastically on Guji_MATH, highlighting the "linguistic tax" of classical Chinese.
As shown in Figure 3, "hidden premises" are essential for solving these problems but are often implicit in the source text. This adds a layer of complexity beyond pure mathematical calculation, requiring models to possess historical and cultural context.
Key Insights and Results
The study revealed several critical findings regarding the capabilities of modern LLMs:
- The "Context Gap": Nearly 23% of model failures were attributed to a lack of historical context, such as misunderstanding ancient metrological standards or unit conversion rules.
- Performance Leaders: DeepSeek R1 achieved the highest accuracy in Closed-Book mode (63.01%), while Qwen-Plus led in Open-Book mode (70.25%).
- Scale and Reasoning: Trillion-parameter models significantly outperformed smaller 32B models, suggesting that scale is vital for navigating low-resource linguistic tasks.
- The Open-Book Boost: Providing the ancient solution significantly improved performance for almost all models, suggesting that while LLMs have the reasoning power, they struggle with the initial comprehension of archaic phrasing.
- Difficulty Scaling: Accuracy dropped sharply as problem complexity increased. For Level 4 problems, even top models struggled to exceed 50% accuracy in Closed-Book settings.
Conclusion
While top-tier models show promise, their performance on Guji_MATH is significantly lower than on modern benchmarks. Classical Chinese's conciseness, where subjects or operators are often omitted, remains a major barrier. Future work aims to expand the dataset to include the "Golden Age" of Chinese mathematics (Song, Yuan, and Ming dynasties) and explore standardization techniques like pseudo-code translation to bridge the comprehension gap.
Summary updated