YRC-Bench: Benchmarking Culturally Aware Machine Translation for Historical Heritage via Tri-Level Semantic Bridging

Huang Yan, Aobo Li, Jiayi Wang, Huidong Zhu

TL;DR

YRC-Bench is a new human-curated benchmark designed to evaluate how well Large Language Models (LLMs) translate Classical Chinese historical heritage into nine diverse target languages. By introducing a "tri-level" architecture that uses Modern Chinese as a semantic bridge, the researchers demonstrate that explicit semantic mediation significantly reduces hallucinations and improves the preservation of culture-specific nuances.

What Problem Does This Paper Address?

Translating historical texts—specifically Classical Chinese (LZH)—is a nightmare for standard machine translation (MT) systems. These texts are characterized by "terse syntax" and "acute semantic asymmetry," where a few characters might encode complex hydraulic engineering techniques or obscure bureaucratic titles.

Current LLMs, while powerful, often fail in two ways when handling this heritage:

  1. Semantic Opacity: They struggle with diachronic shifts (changes in word meaning over time) and omitted subjects/objects common in classical grammar.
  2. Cultural Erasure: They often "domesticate" unique historical terms into generic modern equivalents, losing the specific cultural identity of the source.

Existing benchmarks mostly focus on modern vernacular or simple bilingual conversion (Classical to Modern Chinese), leaving the direct dissemination of historical heritage into foreign languages largely unexamined.

Methods

The authors propose a Tri-Level Parallel Alignment architecture: LZH (Classical) ↔ ZHO (Modern Chinese Gloss) ↔ FOR (Foreign Language).

The Tri-Level Architecture and CSI Operationalization

Key components include:

  • Semantic Bridging: Instead of direct translation, the model uses a Modern Chinese gloss to "unpack" archaic polysemy and resolve syntactic omissions.
  • CSI Operationalization: The researchers annotated Culture-Specific Items (CSIs) and provided two reference variants for each:
  • Conserved (Foreignization): Retains cultural identity via transliteration or historical nomenclature (e.g., "Eastern Sea").
  • Substituted (Domestication): Uses functional equivalents for readability (e.g., "the ocean").
  • Bridging Strategies: They tested models under three settings: Direct Translation (DT), Human-Gloss Bridged (HGB), and Machine-Gloss Bridged (MGB) using Chain-of-Thought prompting.

Datasets

The study introduces YRC-Bench, a high-quality, multi-parallel corpus:

  • Source Material: Authentic narratives from foundational Yellow River treatises like the Shuijing Zhu (Commentary on the Waterways Classic).
  • Scale: 8,500+ evaluation instances and 1,240 unique CSIs.
  • Taxonomy: Covers four domains: Hydraulic Engineering & Geography, Institutional Governance, Material Culture, and Rituals & Customs.
  • Languages: Nine target languages partitioned into Sinographic (Japanese, Korean, Vietnamese) and Non-Sinographic (English, French, German, Spanish, Russian, Arabic) traditions.
Data Distribution of YRC-Bench Dataset

Key Insights and Ideas

  • Modern Chinese as a "Semantic Hub": Modern Chinese (ZHO) acts as a superior intermediate representation because its syntactic structure aligns more closely with the dense pre-training data of modern LLMs. It functions as a "cognitive bridge" that anchors historical fidelity.
  • The Sinographic Divide: There is a profound cross-cultural divergence in translation preferences. Readers in the Sinographic tradition (who share etymological roots with Chinese) prefer Conserved terms because they can intuitively parse them. Non-Sinographic readers generally prefer Substituted terms for accessibility—unless no equivalent exists (the "Forced" condition), in which case they pivot to Conserved terms to avoid factual inaccuracies.
  • Human vs. Machine Glosses: While machine-generated glosses (MGB) help, they are prone to error propagation. If the model misinterprets the classical source initially, the error cascades into the final translation. Human-curated glosses remain the gold standard for high-stakes heritage translation.

Key Results

  • Performance Boost: The Human-Gloss Bridged (HGB) setting yielded absolute COMET score improvements of up to 8.5% over direct translation across all models.
  • Human Audit: In blind expert audits, HGB showed "overwhelming superiority," particularly in rescuing the Culture and Style dimensions, which typically collapse under direct translation.
  • Preference Win-Rates: As shown in Figure 5, the bridged approach (T+M) achieved significantly higher win-rates across all nine target languages compared to direct translation.
Net Human Preference Win-Rate of Bridged over Direct

Limitations and Future Work

  • Scale and Domain: Curation requires rare philological expertise, limiting the dataset's scale. It currently focuses on hydraulic treatises and does not yet cover classical poetry or colloquial historical fiction.
  • Prompt Engineering: The study used standardized zero-shot prompts to control variables; the authors suggest that extensive prompt tuning or few-shot learning could further enhance results.
  • Automation: Future work could explore how to better automate the generation of high-fidelity semantic bridges to reduce the reliance on human experts.

Summary updated

Abstract

Translating historical cultural heritage poses profound challenges for machine translation systems due to the acute semantic asymmetry across eras and cultures, where terse classical syntax frequently obscures domain-specific cultural knowledge. While Large Language Models have demonstrated remarkable prowess in generic translation benchmarks, they struggle to negotiate Culturally-Specific Items and diachronic shifts in domain-rich historical texts, such as Yellow River cultural records. To bridge this critical gap, we introduce YRC-Bench, the first human-curated, multi-parallel benchmark dedicated to historical heritage translation across Classical Chinese, Modern Chinese, and nine target foreign languages. YRC-Bench establishes a tri-level parallel corpus incorporating modern explanatory glosses as explicit semantic bridges and operationalizes translation strategies through paired conserved versus substituted CSI variants categorized by cultural translatability. Systematic evaluations of nine leading LLMs across bidirectional translation directions reveal that leveraging modern Chinese glosses - especially human-curated ones - dramatically elevates both lexical fidelity and semantic adequacy. Furthermore, multidimensional human preference audits by native experts unveil distinct cross-cultural preferences between Sinographic-tradition and non-Sinographic language groups.
BibTeX
@misc{yan2026yrc,
  author = {Huang Yan and Aobo Li and Jiayi Wang and Huidong Zhu},
  title = {{YRC-Bench: Benchmarking Culturally Aware Machine Translation for Historical Heritage via Tri-Level Semantic Bridging}},
  year = {2026},
  eprint = {2611.10555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10555},
}
Figure