Measuring the behavioral impact of machine translation quality improvements with A/B testing

Ben Russell, Duncan GillespieEMNLP 2016 · DOI

Abstract

This paper presents a case study on the impact of domain-specific machine translation (MT) on user behavior within Etsy's global marketplace. While traditional MT evaluation relies on automated metrics like BLEU, these often fail to capture the nuances of user-generated content in specialized domains. We developed and tested three custom English-to-French MT systems. Our findings reveal that a system tuned with a small set of in-domain data (System 3) outperformed systems with higher BLEU scores in human evaluations and real-world A/B testing, leading to an 8.72% increase in purchase conversions. This highlights the importance of domain-specific tuning and behavioral metrics in MT evaluation.

Introduction

In global e-commerce, machine translation is essential for connecting buyers and sellers across language barriers. However, generic MT services often struggle with the "user-generated content" typical of niche marketplaces. For example, the word "clutch" is frequently translated as an automotive part by generic engines, whereas on Etsy, it almost exclusively refers to a handbag. Such errors undermine user trust and hinder sales. This study investigates whether domain-customized MT can improve business KPIs beyond what is suggested by standard NLP metrics.

Methodology

The researchers developed a custom English-to-French MT pipeline consisting of data cleaning, system iteration, and multi-stage evaluation.

  1. Data Cleaning: 130 million product listings were processed using language detection and the Gale-Church alignment algorithm to extract high-quality sentence pairs.
  2. System Iteration: Three distinct systems were trained to test different data and tuning strategies, ranging from maximizing volume to targeted in-domain tuning.
  3. Evaluation: Systems were first evaluated using BLEU scores, followed by human side-by-side comparisons via Amazon Mechanical Turk, and finally a 66-day A/B test on live traffic.
Training data configurations for the three experimental MT systems.

Datasets

The training corpus included a bilingual set of 885,732 aligned English-French pairs and a monolingual set of 2.6 million French segments to improve syntax. System 3 utilized a specialized 2,000-sentence "in-domain" tuning set containing professionally translated Etsy-specific terminology to correct common lexical errors.

Results and Discussion

The automated evaluation showed that System 2 achieved the highest BLEU score (50.36). However, human evaluators preferred System 3, which had the lowest BLEU score (46.85) among the custom systems but utilized the targeted in-domain tuning.

BLEU scores and improvements over the generic baseline for each system.

In the live A/B test, System 3 was compared against a generic baseline. The results demonstrated significant improvements in user engagement and conversion:

  • Purchase Conversion Rate: +8.72%
  • Pages per Visit: +3.37%
  • Add-to-Cart Rate: +2.92%

The discrepancy between BLEU scores and human/behavioral performance suggests that BLEU can be misleading in specialized domains. System 2 likely overfit the automated test sets, whereas System 3's small but high-quality tuning set corrected critical lexical errors that mattered most to users. This study proves that MT quality is a direct lever for business growth.

Conclusion

Customizing MT for specific domains provides substantial value for e-commerce. While automated metrics are useful for initial development, they do not always correlate with user experience. Future work should extend this approach to other language pairs and lower-resource languages. Ultimately, A/B testing should be considered a gold standard for evaluating commercial MT systems.

Summary updated

BibTeX
@inproceedings{russell2016measuring,
  author = {Ben Russell and Duncan Gillespie},
  title = {{Measuring the behavioral impact of machine translation quality improvements with A/B testing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1251},
}
Figure