Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki

Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi, Kosuke Sato, Kentaro Inui, Keisuke Sakaguchi, Benjamin Heinzerling2025 · DOI

TL;DR

Language models (LMs) are heavily biased towards the Gregorian calendar, struggling with non-Gregorian systems like the Japanese "wareki." This paper introduces a comprehensive evaluation of LMs on wareki-related tasks, revealing significant limitations even in Japanese-centric models, and highlighting the need for culturally aware temporal reasoning in AI.

What Problem Does This Paper Address?

Modern language models are trained on vast amounts of text, predominantly in English, leading to an implicit bias towards the Gregorian calendar for temporal reasoning. However, many cultures worldwide use their own calendar systems (e.g., Hijri, Hebrew, Japanese wareki) for daily life, official documents, and cultural practices. The ability of LMs to accurately handle these non-Gregorian calendars is crucial for achieving true cultural competence and enabling reliable NLP applications in diverse linguistic and cultural contexts. This paper specifically addresses the gap in evaluating LM performance on the Japanese wareki system, which is complex due to its era-based structure and irregular year lengths tied to imperial successions.

Methods

The researchers developed three novel evaluation tasks to assess LM capabilities with the Japanese wareki:

  • CALENDARCONVERSION: This task tests the ability to convert dates between the Gregorian calendar and the Japanese wareki system, and vice-versa.
  • JAPANESECALENDARARITHMETIC: This task evaluates temporal reasoning across wareki era boundaries, requiring models to perform calculations (e.g., "ten years after") that span changes in eras.
  • BIRTHYEARRECALL: This task assesses factual recall by asking models to provide the birth year of Japanese individuals in either Gregorian or wareki format.

They evaluated a range of models, including Japanese-centric, English-centric, and frontier models (like GPT-4o and DeepSeek V3), using both English and Japanese prompts. Error analysis was conducted to understand the failure modes.

Datasets

The study introduced synthetic datasets for the three evaluation tasks:

  • CALENDARCONVERSION: Datasets of corresponding Gregorian and wareki years for the Meiji, Taisho, Showa, and Heisei eras.
  • JAPANESECALENDARARITHMETIC: Datasets with unique, non-overlapping dates sampled within a five-year window before and ten years after era transitions, ensuring temporal precision around era boundaries.
  • BIRTHYEARRECALL: Datasets of 300 Japanese individuals per era (Meiji, Taisho, Showa, Heisei) extracted from Wikidata, with birth years provided in both Japanese and English formats.

Key Insights and Ideas

  • Gregorian Bias is Pervasive: Even Japanese-centric LMs, despite being trained on Japanese data, exhibit a significant bias towards the Gregorian calendar, often storing and recalling information in that format.
  • Complexity Matters: While simple date conversions are handled reasonably well by some models, complex tasks like arithmetic across era boundaries or recalling birth years in wareki reveal substantial weaknesses.
  • Era Transitions are Tricky: Models struggle with the irregular nature of wareki era transitions, especially when an era change occurs within a single Gregorian year (e.g., 2019 spanning Heisei 31 and Reiwa 1).
  • Corpus Frequency Plays a Role: The frequency of wareki expressions in the training data appears to correlate with performance on temporal reasoning tasks, suggesting that underrepresentation of non-Gregorian terms hinders accuracy.

Key Results

  • CALENDARCONVERSION: Japanese-centric models, GPT-4o, and DeepSeek V3 achieved near-perfect accuracy in both Gregorian-to-Japanese and Japanese-to-Gregorian conversions. English-centric models showed more variance, with some performing well on simple conversions but struggling with others.
  • JAPANESECALENDARARITHMETIC: A significant performance gap exists between Japanese-centric and English-centric models. While Japanese models showed better performance on recent era transitions (Heisei-Reiwa), even frontier models struggled with older transitions.
  • BIRTHYEARRECALL: All models showed a bias towards the Gregorian calendar. Japanese-centric models performed better than English-centric models when prompted in Japanese, but still underperformed compared to their Gregorian recall accuracy. The "Within ±3 years Match" metric showed that models could often get within a few years of the correct wareki date, suggesting potential reliance on internal Gregorian conversions.
  • CROSSCALENDARCONSISTENCY: Some Japanese-centric models demonstrated high consistency (over 80%) in correctly answering birth years in both wareki and Gregorian formats, while others showed lower consistency (around 50%), indicating variability in how this knowledge is recalled.

Limitations and Future Work

  • Scope: The study focused solely on the Japanese wareki system. Future work should extend this analysis to other non-Gregorian calendars (e.g., Hijri, Hebrew) by designing system-specific tasks that capture their unique temporal structures.
  • Evaluation Method: The current evaluation relies on prompt-based testing and error analysis. Future research could employ probing or mechanistic interpretability methods to directly examine internal representations of calendrical knowledge.
  • Data Availability: Only one Japanese-centric model (llm-jp-3-13b) had publicly released training data. Understanding the distribution of wareki expressions in other models' training data is needed to fully explain performance differences.
  • Model Limitations: The study highlights the need for developing LMs that are better equipped for culture-specific tasks like calendar understanding, suggesting that current models, even those trained on Japanese data, have significant limitations in this area.

Summary updated

BibTeX
@misc{sasaki2025can,
  author = {Mutsumi Sasaki and Go Kamoda and Ryosuke Takahashi and Kosuke Sato and Kentaro Inui and Keisuke Sakaguchi and Benjamin Heinzerling},
  title = {{Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki}},
  year = {2025},
  doi = {10.18653/v1/2025.ijcnlp-short.36},
}
Figure