Machine translation impact in E-commerce multilingual search

Bryan Zhang, Amita MisraEMNLP 2022

TL;DR

This paper introduces a novel framework for evaluating the impact of machine translation (MT) quality on e-commerce multilingual search performance, using behavioral signals instead of human annotations. They propose methods to quantify MT's "launching" and "improvement" impact, identifying language pairs that benefit most from MT investment and guiding strategy choices.

What Problem Does This Paper Address?

Modern e-commerce platforms aim to serve a global audience, meaning users expect to search for products in their native language, even if the product catalog is primarily in another. This necessitates cross-lingual information retrieval (CLIR), where machine translation (MT) plays a crucial role in translating user queries. While previous work suggests a strong correlation between MT quality and CLIR performance, there's a gap in understanding how much MT improvement matters in a real-world, industrial setting, and whether there's a point of diminishing returns. Furthermore, traditional CLIR evaluation often relies on expensive human annotations or pre-annotated datasets that don't reflect the dynamic nature of e-commerce search (e.g., changing product inventories and user behavior). This paper addresses the need for a practical, scalable, and automated way to evaluate MT's impact on e-commerce search and to guide investment in MT systems.

Methods

The authors propose a rank-based evaluation framework that leverages behavioral signals (user clicks and purchases) as a proxy for relevance, eliminating the need for human annotations. This framework works by:

  1. Creating a test set: They sample queries from historical search traffic in the target language and collect associated product IDs and their frequencies (clicks/purchases).
  2. Translating queries: They use both human-translated queries (as a gold standard) and MT-translated queries.
  3. Evaluating search performance: They use the collected behavioral signals to compute traditional relevance metrics like nDCG@16 on the search results obtained from MT-translated queries.

They then introduce two key metrics to quantify MT's impact:

  • MT Launching Impact Rate: This measures the improvement in search metrics (e.g., nDCG) when an MT system is introduced, relative to the improvement in MT quality (e.g., BLEU, Chrf). This helps identify "high-impact" language pairs that are highly sensitive to MT quality.
  • MT Improvement Impact Rate: This quantifies the impact of further improving an MT system (e.g., from a generic MT to a fine-tuned MT) on search performance. This helps determine whether comprehensive MT improvement (CI) or specific improvement (SI) is more beneficial for a given language pair.

Datasets

The paper uses experimental datasets collected from actual e-commerce traffic across 12 language pairs from 6 different stores. For each language pair, they create a test set comprising 4000 queries. The relevance ground truth is derived from user behavioral data (purchased product IDs and their frequencies) associated with these queries.

Key Insights and Ideas

  • Behavioral Signals as a Scalable Proxy: The core innovation is using user clicks and purchases as a reliable proxy for relevance in CLIR evaluation, making it practical for large-scale, continuous evaluation in an industrial setting.
  • Diminishing Returns Exist: The study confirms that improving MT quality doesn't always lead to proportional improvements in search. There's a point where further MT enhancements yield less benefit, especially for certain language pairs.
  • Language Pairs Vary in Sensitivity: Not all language pairs are created equal when it comes to MT impact. Some (e.g., mlin-enin, kokr-enus) are highly sensitive to MT quality ("high-impact"), while others (e.g., ptpt-eses, enus-jajp) are less so ("low-impact").
  • Guiding MT Investment: The proposed metrics help prioritize which language pairs are worth investing in for MT improvement and can guide the type of improvement strategy (comprehensive vs. specific) that would be most effective.
  • Specific Improvement (SI) Can Be Crucial: For some language pairs, targeted improvements (SI) that address specific translation challenges (like brand terms or transliteration) can be more impactful than general MT model upgrades (CI).

Key Results

  • The proposed framework successfully creates e-commerce CLIR test sets at scale without human annotation.
  • They identified "high-impact" language pairs (e.g., mlin-enin, kokr-enus) where MT launching has a significant positive effect on search results.
  • For mlin-enin, a 1-point increase in BLEU score can lead to a similar increase in search improvement, indicating high sensitivity.
  • For kokr-enus, while MT launching has a good impact, the MT improvement rate is lower, suggesting specific improvements (SI) might be more beneficial than comprehensive ones (CI).
  • A/B testing results largely corroborated the offline evaluation findings, showing significant business metric improvements (Order Product Sales, composite contribution profit) for high-impact language pairs with improved MT systems.
  • For ptpt-eses and nlnl-dede, specific MT improvements targeting 3-5% of query traffic showed a greater impact than comprehensive improvements.

Limitations and Future Work

  • MT Evaluation Metrics: The paper notes that while BLEU and Chrf are used, they might not perfectly capture the nuances of query translation for search. They suggest that a general-purpose MT metric might not be ideal for this specific downstream task.
  • Domain Specificity: While they use domain-specific MT models, the evaluation framework itself is designed to be adaptable to different e-commerce domains.
  • Future Work: The authors suggest that their framework can be used to create more e-commerce CLIR test sets at scale. They also imply that further research could explore more sophisticated MT evaluation metrics tailored for search, and investigate the interplay of MT with other components of the search pipeline.

Summary updated

BibTeX
@inproceedings{zhang2022machine,
  author = {Bryan Zhang and Amita Misra},
  title = {{Machine translation impact in E-commerce multilingual search}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure