Measuring the Effects of Human and Machine Translation on Website Engagement

Geza Kovacs, John DeNeroAMTA 2022 · OpenReview

Introduction

Localization is a critical challenge for global web publishers. While professional human translation (HT) offers high quality at a high cost, machine translation (MT) provides a scalable but potentially lower-quality alternative. This study presents a massive "in-the-wild" A/B test involving over 3.3 million web sessions to quantify how these strategies impact real-world user engagement.

Methodology

The researchers conducted large-scale A/B tests on the homepages of two open-source projects: UNetbootin (Site 1) and HabitLab (Site 2). Visitors with non-English browser settings were randomized into five conditions:

  • UE (Untranslated English): The control group; no translation provided.
  • HTT (Human Translation, default): Users see a professional translation immediately.
  • HTE (Human Translation, English default): Users see English but can manually switch to the human translation.
  • MTT (Machine Translation, default): Users see a Google Translate version immediately.
  • MTE (Machine Translation, English default): Users see English but can manually switch to the MT version.

Engagement was measured through specific user actions, including download clicks, non-download clicks, scrolling behavior, and visit duration.

The "Switcher" Fallacy

A pivotal finding of the study is that users rarely interact with manual language switchers. Even when the content is in a language they may struggle to read, only about 2% of users in the HTE and MTE groups manually opted to switch to their preferred language.

Manual language switching rates across conditions.

This behavior suggests that if a translation is not provided by default, the vast majority of international users will never see it, regardless of the translation's quality. If you don't show the translation immediately, you effectively lose the benefit of having it.

Impact on Engagement

The data reveals that providing any form of translation significantly boosts engagement compared to raw English. On Site 1 (UNetbootin), both human and machine translations (HTT and MTT) increased the proportion of users who scrolled and stayed on the page for longer durations.

Engagement metrics for Site 1 (UNetbootin) across experimental conditions.

While human translation remains the gold standard for driving "deep" engagement—such as clicking text links and scrolling—machine translation proved to be a highly effective "good enough" solution. MTT significantly outperformed the untranslated English control (UE) in keeping users on the page. Interestingly, the "Download Paradox" was observed on Site 1, where users were slightly more likely to click the prominent download button on the English page, likely because the button's function is visually intuitive and requires minimal reading.

The Role of In-Browser Translation

In the absence of site-provided translations (the UE condition), a significant minority of users (roughly 5–10%) relied on built-in browser tools, such as Chrome's "Translate this page" feature. This reliance was most pronounced among users from regions with lower English proficiency, highlighting that users will seek out translation tools as a safety net when publishers fail to provide localized content.

Conclusion

The study suggests a clear hierarchy for web localization: Auto-detected Human Translation > Auto-detected Machine Translation > English. The most critical takeaway for developers is to avoid forcing users to find a language toggle; automatic detection and delivery of translated content is essential for retaining international audiences. While HT is superior for text-heavy engagement, MT is a powerful and necessary tool for global reach.

Summary updated

BibTeX
@inproceedings{kovacs2022measuring,
  author = {Geza Kovacs and John DeNero},
  title = {{Measuring the Effects of Human and Machine Translation on Website Engagement}},
  booktitle = {AMTA},
  year = {2022},
  url = {https://openreview.net/forum?id=JOgrEEk3RUY},
}
Figure