Abstract
This study evaluates the English subtitle translation performance of two large language models, DeepSeek and Moonshot AI, for the documentary China Season 2 from the perspective of cultural translation. By collecting 875 subtitle data containing culture-loaded words, poems, and terms, and combining word frequency analysis, BLEU/ROUGE automated scoring, and in-depth case analysis, it is found that the two models show 63% consistency at the lexical semantic level, but differ significantly in phrasal structure and cultural strategy selection. DeepSeek excels in literal translation retention of calendrical terms such as "sui/si/zai" and rhythmic reconstruction of Du Fu's poems, while Moonshot AI has an advantage in cultural interpretation of metaphors like "the smell of wine and meat from the vermilion gates" and contextual coherence of Li Bai's image. The study reveals problems such as semantic deviation and format norm defects in AI-based cultural subtitle translation, providing an empirical basis for constructing a "technology-culture" two-dimensional evaluation framework.
Introduction
As China expands its cultural outreach through high-production documentaries, the "subtitle bottleneck" remains a significant hurdle. Historical documentaries are packed with "culture-loaded" words—idioms, ancient bureaucratic titles, and classical poetry—that often lose their meaning when translated literally. Traditional machine translation (MT) often distorts these nuances. While Large Language Models (LLMs) offer better reasoning, there is a lack of rigorous frameworks to evaluate how they handle the "technology-culture" divide. This study benchmarks two leading models, DeepSeek and Moonshot AI, against the culturally dense subtitles of the documentary China, revealing a fundamental strategic divide in how AI handles historical "schemas."
Methodology
The researchers employed a hybrid evaluation model that blends computational linguistics with expert manual analysis. Key components include:
- The "Three-Dimensional, Nine-Category" Framework: A novel classification system that categorizes cultural elements into Historical Systems (e.g., calendrical terms), Literary Imagery (e.g., metaphors), and Religious/Philosophical Concepts (e.g., Taoist terms).
- Automated Metrics: Standard BLEU and ROUGE scores were used to assess lexical similarity and fluency.
- Word Frequency & Co-occurrence Analysis: Using tools like KhCoder to visualize how models associate cultural concepts (e.g., how strongly "Li Bai" is linked to "Chang'an").
- Manual Annotation: Three translation experts performed double-blind coding to evaluate "Strategy Choice" (Literal vs. Free translation), "Cultural Loss," and "Format Norms."
- Cultural Adaptation Index (CAI): A proposed composite metric that integrates automated scores with manual assessments of fidelity and acceptability.
Strategic Divergence: Foreignization vs. Domestication
The models exhibit distinct "personalities" in their translation choices. DeepSeek leans toward foreignization, preserving cultural symbols and literal terms. For example, it maintains ancient year-designations like sui/si/zai. Conversely, Moonshot AI leans toward domestication, prioritizing the English reader's experience through "free translation" and explanatory additions.
This divergence is most visible in poetic translation. DeepSeek achieved a 57% success rate in preserving end-rhymes in five-character quatrains. Moonshot AI, while struggling with rhyme, excelled at "enjambment"—making the lines flow more naturally as a single thought for a modern audience.
Case Study: The "Arched Trees" Failure
A standout example of the models' limitations is the idiom mù mù yǐ gǒng (墓木已拱). Literally "the tomb trees have arched," it signifies that a person has been dead so long that the trees on their grave are now thick enough to require two people to encircle them with their arms (gǒng). DeepSeek translated this literally, missing the temporal meaning entirely. Moonshot inferred the passage of time correctly but lost the vivid imagery. Both models often fail at the "why" level—they can translate what a term is but fail to supplement the cultural connotation that makes the term significant to a historian.
Quantitative Results
The quantitative analysis highlights the difficulty of cultural translation even for advanced AI:
- Lexical Consistency: The two models showed only 63% consistency at the lexical level, indicating widely different approaches to word choice.
- Performance Drop: The BLEU-4 score for segments containing culture-loaded words was only 0.29, significantly lower than the 0.47 achieved for ordinary text.
- Literal Retention: DeepSeek had an 89% literal translation retention rate for calendrical terms.
- Acceptability: Moonshot AI employed free translation strategies at a rate of 52% (vs. DeepSeek's 34%), leading to higher "acceptability" scores among human evaluators.
- Error Rates: The study found a 14.7% mistranslation rate for cultural concepts across both models.
Conclusion: A Collaborative Future
The study reveals that while LLMs have revolutionized translation, they still struggle with the "cultural discount" found in high-stakes media. To address these limitations, the authors propose a Human-AI Collaborative Workflow:
- AI Initial Translation: Handles 70% of standardized content and terminology consistency.
- Translator Polishing: Focuses on culture-loaded words and poetic imagery.
- Cultural Expert Review: Verifies deep conceptual accuracy and historical context.
This "three-level" approach aims to leverage AI's efficiency while preserving the "soul" of the cultural narrative.
Summary updated