Abstract
This paper presents a case study on the impact of domain-specific machine translation (MT) on user behavior within Etsy's global marketplace. While traditional MT evaluation relies on automated metrics like BLEU, these often fail to capture the nuances of user-generated content in specialized domains. We developed and tested three custom English-to-French MT systems. Our findings reveal that a system tuned with a small set of in-domain data (System 3) outperformed systems with higher BLEU scores in human evaluations and real-world A/B testing, leading to an 8.72% increase in purchase conversions. This highlights the importance of domain-specific tuning and behavioral metrics in MT evaluation.
Introduction
In global e-commerce, machine translation is essential for connecting buyers and sellers across language barriers. However, generic MT services often struggle with the "user-generated content" typical of niche marketplaces. For example, the word "clutch" is frequently translated as an automotive part by generic engines, whereas on Etsy, it almost exclusively refers to a handbag. Such errors undermine user trust and hinder sales. This study investigates whether domain-customized MT can improve business KPIs beyond what is suggested by standard NLP metrics.
Methodology
The researchers developed a custom English-to-French MT pipeline consisting of data cleaning, system iteration, and multi-stage evaluation.
- Data Cleaning: 130 million product listings were processed using language detection and the Gale-Church alignment algorithm to extract high-quality sentence pairs.
- System Iteration: Three distinct systems were trained to test different data and tuning strategies, ranging from maximizing volume to targeted in-domain tuning.
- Evaluation: Systems were first evaluated using BLEU scores, followed by human side-by-side comparisons via Amazon Mechanical Turk, and finally a 66-day A/B test on live traffic.
Datasets
The training corpus included a bilingual set of 885,732 aligned English-French pairs and a monolingual set of 2.6 million French segments to improve syntax. System 3 utilized a specialized 2,000-sentence "in-domain" tuning set containing professionally translated Etsy-specific terminology to correct common lexical errors.
Results and Discussion
The automated evaluation showed that System 2 achieved the highest BLEU score (50.36). However, human evaluators preferred System 3, which had the lowest BLEU score (46.85) among the custom systems but utilized the targeted in-domain tuning.
In the live A/B test, System 3 was compared against a generic baseline. The results demonstrated significant improvements in user engagement and conversion:
- Purchase Conversion Rate: +8.72%
- Pages per Visit: +3.37%
- Add-to-Cart Rate: +2.92%
The discrepancy between BLEU scores and human/behavioral performance suggests that BLEU can be misleading in specialized domains. System 2 likely overfit the automated test sets, whereas System 3's small but high-quality tuning set corrected critical lexical errors that mattered most to users. This study proves that MT quality is a direct lever for business growth.
Conclusion
Customizing MT for specific domains provides substantial value for e-commerce. While automated metrics are useful for initial development, they do not always correlate with user experience. Future work should extend this approach to other language pairs and lower-resource languages. Ultimately, A/B testing should be considered a gold standard for evaluating commercial MT systems.
Summary updated