LLMs for Translation: Historical, Low-Resourced Languages and Contemporary AI Models

Merve TekgürlerNAACL 2025

TL;DR

This paper investigates how Large Language Models (LLMs), specifically Google's Gemini 1.5 Pro, handle the translation of historical, low-resourced languages, using an 18th-century Ottoman Turkish manuscript as a case study. The study reveals that while LLMs show promise, their built-in AI safety mechanisms can inadvertently block and prevent the translation of sensitive historical content, hindering accurate and complete translations.

What Problem Does This Paper Address?

Machine Translation (MT) has seen significant advancements with LLMs, but their application to historical and low-resourced languages, which often lack extensive digital data, remains a challenge. Ottoman Turkish (OT) is a prime example of such a language, making translation a labor-intensive task with limited scholarly resources. Furthermore, mainstream MT research often overlooks the unique challenges posed by these languages. This paper also delves into the ethical implications of AI safety mechanisms within LLMs, specifically how these protocols, designed to mitigate harm, might impede the translation of complex and sensitive historical narratives. The authors aim to understand if these safety mechanisms are a result of the manuscript's content or the language's low-resource status.

Methods

The study employs a quantitative analysis of Google's Gemini 1.5 Pro's translation performance on an 18th-century Ottoman Turkish manuscript. The core methodology involves:

  • Sentence-level Translation and Analysis: The Ottoman Turkish manuscript and its English translation were aligned at the sentence level. Gemini 1.5 Pro was then used to translate the manuscript sentence by sentence.
  • Safety Mechanism Evaluation: The researchers meticulously recorded which sentences were blocked by Gemini's safety mechanisms and the reasons for these blocks (e.g., hate speech, dangerous content, harassment, sexually explicit).
  • Comparative Analysis: The same analysis was performed on a German translation of the same manuscript to compare how Gemini handled the content in a higher-resourced language versus the low-resourced OT.
  • Quantitative Metrics: The study reports on the percentage of untranslated sentences and analyzes the relationship between the severity and probability scores assigned by the safety mechanisms.

Datasets

The primary dataset used for evaluation is:

  • Ottoman Turkish-English (OT-EN) Parallel Text: This dataset consists of 757 sentence pairs derived from the 18th-century Ottoman Turkish manuscript "Prisoner of the Infidels: The Memoirs of Osman Agha of Timișoara" and its English translation by Giancarlo Casale (2021).
  • German-English (DE-EN) Parallel Text: A dataset of 1,699 sentence pairs from the German translation of the same manuscript (by Kreutel and Spies, 1954) and its English translation was also used for comparative analysis.

The paper also mentions preliminary experiments with other LLMs (GPT-3.5, GPT-4, Gemini 1.0, Cohere Aya, GPT-40, GPT-03-mini, Gemini 2.0, Claude Sonnet 3.7) and a fine-tuned Helsinki NLP Opus NMT model for Turkish-English translation.

Key Insights and Ideas

  • Safety Mechanisms Can Be Overly Cautious: The study highlights that LLM safety filters, while intended to prevent harm, can be overly sensitive and block legitimate historical content, especially when dealing with sensitive topics like warfare, violence, or sexual assault, even if not explicitly pornographic.
  • Language Resource Status vs. Content: The comparative analysis between OT-EN and DE-EN translations suggests that the flagging of sentences is more strongly related to the content of the manuscript rather than solely to Ottoman Turkish being a low-resourced language.
  • Nuance and Context are Lost: The blocking of sentences prevents the LLM from providing a complete and accurate translation, potentially silencing the voices of historical figures and obscuring complex narratives.
  • Ethical Implications for Historical Research: The research underscores the need to understand how AI safety mechanisms impact the accessibility and interpretation of historical texts, especially those dealing with difficult human experiences.
  • Translation is Not Just Chat: The paper argues that treating translation as equivalent to general chat interactions can lead to misapplication of safety protocols, as translation often requires rendering sensitive content accurately for specific purposes (e.g., legal, historical documentation).

Key Results

  • Significant Untranslated Passages: For the OT-EN translation, approximately 23% of the manuscript (174 out of 755 sentences) remained untranslated due to safety flags. For the DE-EN translation, 19% (327 out of 1699 sentences) were untranslated.
  • Severity and Confidence Correlation: A strong positive correlation (coefficients of 0.914 for OT and 0.935 for DE) was observed between the severity score of a flagged passage and the model's confidence in its assessment.
  • Category-Specific Trends: The model showed less confidence in classifying "dangerous content" and more confidence in classifying "harassment."
  • Specific Examples of Misclassification: The paper details specific instances where sentences were flagged for "sexually explicit" content due to word usage with different meanings in historical vs. modern Turkish (e.g., "kerhâne"), or for "harassment" and "hate speech" when describing historical violence.
  • Age Sensitivity: A specific test case revealed that Gemini's safety settings were triggered by a sentence describing sexual assault in conjunction with a young age, even when the safety settings were turned off, suggesting deeply embedded filters.

Limitations and Future Work

  • Limitations:
    • The study focused on a single LLM (Gemini 1.5 Pro) and a single historical manuscript.
  • The exact implementation and reasoning behind Gemini's safety algorithms are not fully transparent.
  • While the authors argue content is the primary driver of flagging, the interaction between content and language resource status is complex.
  • Future Work:
    • Investigating how LLMs handle age in sensitive contexts.
  • Further research into the ethical implications of AI safety mechanisms on historical narratives and the potential for silencing marginalized voices.
  • Developing more nuanced AI safety protocols that can distinguish between the mention and use of harmful content in translation.
  • Exploring methods to mitigate the impact of safety settings on low-resourced language translation without compromising genuine safety.

Summary updated

BibTeX
@inproceedings{tekgurler2025llms,
  author = {Merve Tekgürler},
  title = {{LLMs for Translation: Historical, Low-Resourced Languages and Contemporary AI Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure