An Empirical Accuracy Law for Sequential Machine Translation: the Case of Google Translate

Lucas Nunes Sequeira, Bruno Moreschi, Fábio Gagliardi Cozman, Bernardo FontesarXiv 2020 · arXiv

Abstract

As global digital infrastructure increasingly relies on automated agents to relay information across borders, understanding the stability of sequential translation chains becomes critical. This study investigates the degradation of semantic meaning and text size when a source text is subjected to up to 284 consecutive translation "hops" using Google Translate. We propose an Accuracy Empirical Law (AEL) based on a power law decay model and demonstrate that translation quality is heavily influenced by the linguistic distance between languages in the chain.

Introduction

In the classic game of "Chinese Whispers," a message passed through a sequence of intermediaries inevitably morphs into something unrecognizable. In the context of Neural Machine Translation (NMT), a similar phenomenon occurs. While NMT has achieved high performance for single-hop translations, the holistic behavior of translation chains—where text moves from Language A to B, then B to C, and so on—remains poorly understood. This paper quantifies this degradation, seeking to determine if there is a mathematical regularity to how errors compound and how linguistic ancestry affects this decay.

Methodology

The researchers utilized the Google Translate API to construct three types of translation chains:

  1. Randomized: 71 languages selected from Google’s 103 available languages, ordered randomly.
  2. Common: Languages with close common ancestry (e.g., the Neolatine group: Italian, Portuguese, Spanish).
  3. Mixed: Languages from widely divergent families (e.g., Slavic, Sino-Tibetan, Germanic).

The primary metric used was the GLEU (Google Bilingual Evaluation Understudy) score, which measures the minimum of recall and precision of n-grams between the output and the original source. Four seed texts were used: two short excerpts (t1, t3) and two longer excerpts (t2, t4) in English and Portuguese.

The Accuracy Empirical Law (AEL)

The study's central finding is that accuracy loss follows a predictable power law: $$AEL(t) = (t + 1)^{-\alpha}$$ where $t$ represents the number of translation hops and $\alpha$ is the semantic divergence factor. The model fit the empirical data with a remarkably low Root Mean Square Error (RMSE) of 0.02 to 0.04.

GLEU accumulated score decay over sequential translation hops.

The divergence factor $\alpha$ varies significantly based on linguistic proximity. For "common" language groups, $\alpha$ was measured at 0.110, whereas for "mixed" groups, it increased to 0.290—nearly triple the rate of decay.

The "Textual Black Hole" Phenomenon

Beyond semantic accuracy, the researchers observed a consistent reduction in text size, a phenomenon they termed the "textual black hole." As translations progress, the AI tends toward brevity through fragmentation (omission of connectors), incoherent syntax, and literalism.

Text size reduction in common (top) versus mixed (bottom) translation chains.

In one extreme case, a 258-word text was reduced to just seven words after 284 random hops: "Read the Bible and forget it." This suggests that NMT systems have an inherent bias toward summarization when faced with compounding uncertainty.

Conclusion

The discovery of the Accuracy Empirical Law provides a mathematical framework for predicting information loss in automated relay systems. The results underscore the importance of linguistic genealogy; translating within the same language family preserves meaning significantly longer than jumping between unrelated families. Future work may involve applying this law to optimize translation paths, potentially identifying "bridge" languages that minimize semantic divergence.

Reference: Sequeira, L. N., Moreschi, B. S., Cozman, F. G., & Fontes, B. (2020). An Empirical Accuracy Law for Sequential Machine Translation: the Case of Google Translate. arXiv:2003.02817v2 [cs.CL].

Summary updated

Abstract

In this research, we have established, through empirical testing, a law that relates the number of translating hops to translation accuracy in sequential machine translation in Google Translate. Both accuracy and size decrease with the number of hops; the former displays a decrease closely following a power law. Such a law allows one to predict the behavior of translation chains that may be built as society increasingly depends on automated devices.
BibTeX
@misc{sequeira2020empirical,
  author = {Lucas Nunes Sequeira and Bruno Moreschi and Fábio Gagliardi Cozman and Bernardo Fontes},
  title = {{An Empirical Accuracy Law for Sequential Machine Translation: the Case of Google Translate}},
  year = {2020},
  eprint = {2003.02817},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.02817},
}
Figure