Long to reign over us: A Case Study of Machine Translation and a New Monarch

Rebecca Knowles, Samuel LarkinACL 2023 · DOI

Abstract

Machine translation (MT) systems are heavily reliant on historical training data, which can lead to significant failures when real-world terminology shifts abruptly. This paper examines the transition from Queen Elizabeth II to King Charles III as a case study in "world knowledge shifts." Focusing on the English-French language pair, the study reveals how decades of data bias toward "Her Majesty" cause even state-of-the-art systems to produce gender-incorrect translations despite clear contextual clues.

Introduction: The Challenge of World Knowledge Shifts

Machine translation systems are only as robust as the corpora they ingest. For seventy years, "The Queen" was the primary monarch of the Commonwealth, leading to a massive imbalance in bilingual datasets like the Canadian Hansard. This historical bias creates a "stale knowledge" problem: when the world changes—such as the accession of a new King—MT systems often fail to adapt, defaulting to the most statistically probable historical patterns.

This issue is particularly acute in English-French translation due to grammatical gender asymmetries. While English possessive pronouns (His/Her) must agree with the referent's gender, the French equivalent Sa Majesté is grammatically feminine regardless of the monarch's gender.

Methodology and Datasets

The researchers evaluated three MT systems:

  1. Google Translate (Public)
  2. Bing Translator (Public, French-Canada)
  3. Internal System: A Transformer-based model trained on 5.6 million lines of the Canadian Hansard (2006–2021).

The evaluation used a "miniature challenge set" including the Prime Minister’s official announcement of King Charles III’s accession, the Royal Anthem, and recent held-out Hansard records. The training data for the internal system highlighted the scale of the bias: it contained 882 instances of "Her Majesty" compared to only 36 of "His Majesty."

Linguistic Ambiguity and Data Bias

The core difficulty lies in the "one-to-many" mapping between French and English. As shown in the following figure, certain terms are inherently ambiguous without document-level context.

Linguistic ambiguities and context requirements for royal terminology.

For example, while "His Majesty" translates unambiguously to Sa Majesté in French, the reverse requires coreference resolution to determine if the referent is male or female. If a system has "memorized" that Sa Majesté almost always refers to a Queen, it will override local context—even if "King Charles" is mentioned in the same sentence.

Key Results: The "Her Majesty" Default

The results demonstrate that while public systems have some mechanisms to handle these shifts, they are not immune to errors. The internal system, being purely data-driven from historical records, struggled significantly.

Translation accuracy across systems for royal accession texts.

Key findings include:

  • Internal System Failure: On the French-to-English translation of the Prime Minister's announcement, the internal system correctly translated only 17 out of 24 terms, frequently defaulting to "Her Majesty."
  • Subword Issues: The internal system's vocabulary was so biased that the word roi (king) was segmented into individual characters ('r', 'o', 'i') because it appeared too infrequently in the training set to be recognized as a single token.
  • Contextual Overrides: In the Hansard test set, all three systems translated Sa Majesté as "Her Majesty" in 100% of cases where context was required to disambiguate, even when the surrounding text referred to the new King.

Conclusion

This case study underscores the limitations of static MT models in a changing world. The "Her Majesty" default is a clear example of how statistical weight can override explicit contextual clues. Future research must focus on methods for rapid terminology updates and improved document-level context handling, allowing models to "unlearn" outdated biases without requiring full retraining cycles.

Summary updated

BibTeX
@inproceedings{knowles2023long,
  author = {Rebecca Knowles and Samuel Larkin},
  title = {{Long to reign over us: A Case Study of Machine Translation and a New Monarch}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.412},
}
Figure